Nov 02 14:53:11.027474 ip-10-0-133-143 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 14:53:11.027487 ip-10-0-133-143 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 14:53:11.027497 ip-10-0-133-143 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 14:53:11.027902 ip-10-0-133-143 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 14:53:21.221984 ip-10-0-133-143 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 14:53:21.222006 ip-10-0-133-143 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9507a8b162d34748810dac364446f5d1 --
Nov 02 14:55:05.597617 ip-10-0-133-143 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:55:06.041950 ip-10-0-133-143 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:06.041950 ip-10-0-133-143 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:55:06.041950 ip-10-0-133-143 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:06.041950 ip-10-0-133-143 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:06.048843 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048778    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.048843 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048842    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.048843 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048845    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048848    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048851    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048854    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048856    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048859    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048862    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048865    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048870    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048873    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048876    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048878    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048881    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048889    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048894    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048899    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048902    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048904    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.048931 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048907    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048909    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048912    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048915    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048918    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048920    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048923    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048925    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048928    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048931    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048933    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048936    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048939    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048942    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048945    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048948    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048950    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048953    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048956    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048958    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.049382 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048961    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048963    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048965    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048968    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048970    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048973    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048976    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048978    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048981    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048983    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048986    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048988    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048991    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048993    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048996    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.048999    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049003    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049008    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049012    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.049887 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049015    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049018    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049021    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049024    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049026    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049029    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049031    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049034    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049037    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049039    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049042    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049044    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049048    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049051    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049053    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049056    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049058    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049061    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049063    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.050330 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049066    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049068    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049071    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049073    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049076    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049079    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049081    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049084    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049087    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049090    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049093    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049096    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049099    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049102    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049104    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049107    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049109    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049112    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049114    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049117    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.050793 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049119    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049122    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049124    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049127    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049130    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049132    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049138    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049875    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049878    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049881    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049884    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049887    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049890    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049892    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049896    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049898    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049901    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049904    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.051279 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049906    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049909    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049912    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049915    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049917    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049920    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049922    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049925    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049928    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049930    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049933    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049936    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049938    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049941    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049943    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049946    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049948    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049951    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049953    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049956    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.051753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049960    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049962    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049966    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049968    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049971    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049973    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049976    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049979    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049981    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049983    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049986    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049988    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049991    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049994    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.049997    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050000    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050002    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050005    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050007    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050010    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.052225 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050012    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050015    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050019    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050023    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050026    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050029    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050032    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050034    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050037    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050040    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050042    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050045    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050048    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050051    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050054    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050056    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050059    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050061    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050064    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050066    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.052710 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050069    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050072    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050074    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050077    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050079    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050083    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050088    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050090    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050093    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050095    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050098    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050100    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050102    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050105    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050108    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050113    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050116    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.053183 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050118    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050121    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050123    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050126    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050128    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050132    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050136    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050148    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050151    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050153    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050156    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050158    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050161    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050163    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050166    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050671    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050677    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050679    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050683    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050685    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.053663 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050688    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050690    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050693    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050696    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050698    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050701    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050703    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050706    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050708    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050714    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050717    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050720    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050723    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050725    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050728    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050730    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050733    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050735    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050743    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.054133 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050746    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050749    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050751    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050754    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050756    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050758    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050761    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050763    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050766    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050768    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050771    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050774    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050776    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050779    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050781    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050784    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050786    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050789    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050792    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.054612 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050794    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050797    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050799    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050802    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050805    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050809    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050812    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050815    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050818    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050821    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050823    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050826    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050828    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050836    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050839    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050842    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050844    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050847    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050849    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050852    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.055103 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050854    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050857    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050859    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050863    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050865    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050868    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050871    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050874    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050876    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050879    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050881    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050884    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050886    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050889    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050892    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050895    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050897    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050900    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050902    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050905    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.055623 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050907    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050910    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050912    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050914    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050917    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050919    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050927    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050930    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050933    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050935    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050938    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050941    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050943    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050945    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050948    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050953    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050957    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050960    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050963    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.056093 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050965    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050968    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.050970    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.051988    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.051998    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052005    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052009    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052016    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052019    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052023    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052028    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052031    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052034    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052037    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052040    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052043    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052046    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052049    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052052    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052055    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052065    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052072    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052075    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:55:06.056585 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052078    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052081    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052085    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052089    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052092    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052095    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052098    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052101    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052104    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052107    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052110    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052113    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052117    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052120    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052123    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052126    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052131    2577 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052134    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052140    2577 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052143    2577 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052146    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052149    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052152    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052156    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:55:06.057122 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052159    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052162    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052165    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052168    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052171    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052174    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052183    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052186    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052189    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052192    2577 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052196    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052199    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052202    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052205    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052208    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052211    2577 flags.go:64] FLAG: --help="false"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052214    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052217    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052220    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052223    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052227    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052230    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052233    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:55:06.057727 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052236    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052241    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052244    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052247    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052250    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052253    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052256    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052259    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052262    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052265    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052268    2577 flags.go:64] FLAG: --lock-file=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052271    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052274    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052277    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052282    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052285    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052296    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052299    2577 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052302    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052305    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052308    2577 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052311    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052316    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052319    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052323    2577 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:55:06.058297 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052326    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052329    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052332    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052336    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052339    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052341    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052344    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052351    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052355    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052359    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052373    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052376    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052379    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052382    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052385    2577 flags.go:64] FLAG: --port="10250"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052388    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052390    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03964f7dc41bb8368"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052394    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052397    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052400    2577 flags.go:64] FLAG: --register-node="true"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052403    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052406    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052409    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052412    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052421    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052425    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:55:06.058907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052428    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052431    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052434    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052439    2577 flags.go:64] FLAG: --runonce="false"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052442    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052445    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052448    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052451    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052453    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052456    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052459    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052462    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052465    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052468    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052472    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052476    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052479    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052482    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052485    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052490    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052493    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052496    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052502    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052505    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052508    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:55:06.059550 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052511    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052514    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052517    2577 flags.go:64] FLAG: --v="2"
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052521    2577 flags.go:64] FLAG: --version="false"
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052525    2577 flags.go:64] FLAG: --vmodule=""
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052529    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.052538    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052709    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052713    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052716    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052720    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052723    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052725    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052728    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052732    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052736    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052739    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052742    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052744    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052748    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052751    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.060187 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052754    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052756    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052760    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052762    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052765    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052767    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052770    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052772    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052775    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052778    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052780    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052783    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052785    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052788    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052790    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052793    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052795    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052798    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052801    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.060695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052803    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052806    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052809    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052812    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052814    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052817    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052819    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052823    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052827    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052829    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052832    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052835    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052837    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052840    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052843    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052845    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052848    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052850    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052853    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052856    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.061861 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052858    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052860    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052863    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052865    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052868    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052870    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052873    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052875    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052878    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052880    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052883    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052885    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052888    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052891    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052895    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052897    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052900    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052903    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052905    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052908    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.062334 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052910    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052913    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052915    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052917    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052920    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052922    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052925    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052927    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052930    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052932    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052935    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052938    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052940    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052943    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052945    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052947    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052950    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052952    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052955    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052957    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.062815 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052960    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052962    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052965    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052967    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052970    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052973    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052979    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052982    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052985    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052987    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052989    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.052992    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.053019    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053052    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053056    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.063301 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053059    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053062    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053064    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053066    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053069    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053072    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053075    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053078    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053080    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053083    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053088    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053090    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053093    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053095    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053098    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053100    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053103    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053105    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.063695 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053108    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053112    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053114    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053119    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053122    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053124    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053127    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053129    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053132    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053136    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053140    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053143    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053146    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053149    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053151    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053154    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053156    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053159    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053161    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053164    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.064135 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053167    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053169    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053172    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053174    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053177    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053180    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053182    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053184    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053187    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053190    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053193    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053195    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053198    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053202    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053205    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053208    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053211    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053215    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053218    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053220    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.064678 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053223    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053225    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053228    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053230    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053233    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053235    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053238    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053240    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053243    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053246    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053248    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053251    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053253    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053256    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053258    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053261    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053263    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053266    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053268    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053271    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.065145 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053273    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053276    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053278    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053281    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053284    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053288    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053290    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053294    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053296    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053300    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053302    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053305    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053307    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053310    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053312    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053315    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053317    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053320    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053323    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053325    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.065631 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053328    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053330    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053335    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053338    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.053341    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.059811    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.059824    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.060569    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.060577    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060635    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060640    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060644    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060647    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060650    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060653    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060656    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060659    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060661    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060664    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.066138 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060667    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060669    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060672    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060675    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060678    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060681    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060684    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060687    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060689    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060692    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060694    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060697    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060700    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060702    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060705    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060708    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060710    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060713    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060716    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060719    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.066589 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060722    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060725    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060728    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060730    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060733    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060735    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060738    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060741    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060744    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060746    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060749    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060751    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060754    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060756    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060759    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060765    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060767    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060770    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060773    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060775    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.067048 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060778    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060781    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060783    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060786    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060789    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060791    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060793    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060796    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060798    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060801    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060804    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060807    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060809    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060812    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060814    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060817    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060820    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060823    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060825    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060828    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.067596 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060831    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060833    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060836    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060838    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060841    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060843    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060846    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060848    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060851    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060854    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060856    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060858    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060861    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060864    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060867    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060869    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060872    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060874    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060877    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060879    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.068058 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060882    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060886    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060890    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060893    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060896    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060898    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060901    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060904    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060906    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060910    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060914    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060917    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060920    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060922    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060925    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.068533 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.060927    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.060955    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061079    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061083    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061087    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061089    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061092    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061095    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061098    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061100    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061103    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061106    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061108    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061111    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061113    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061116    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.069020 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061118    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061121    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061123    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061126    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061129    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061131    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061134    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061136    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061139    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061141    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061144    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061146    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061149    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061151    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061154    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061156    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061159    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061161    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061165    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.069659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061169    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061171    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061174    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061177    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061180    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061182    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061185    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061188    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061190    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061193    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061195    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061198    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061200    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061203    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061206    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061208    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061210    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061213    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061216    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061218    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.070385 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061221    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061223    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061226    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061228    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061231    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061233    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061236    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061240    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061244    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061247    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061250    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061253    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061255    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061258    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061261    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061263    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061266    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061269    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061272    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.071109 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061275    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061277    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061280    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061282    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061285    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061287    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061290    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061292    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061295    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061297    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061300    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061302    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061305    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061307    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061310    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061312    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061315    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061317    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061319    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061322    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.071915 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061324    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061327    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061329    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061332    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061334    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061336    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061339    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061341    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061344    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061346    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061349    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061352    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061355    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061358    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.061392    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061423    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:06.072493 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061428    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061431    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061434    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061437    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061440    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061442    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061444    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061447    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061450    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061453    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061455    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061458    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061461    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061463    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061466    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061468    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061470    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061473    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061475    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061478    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:06.072869 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061480    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061483    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061485    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061488    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061490    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061492    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061495    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061497    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061504    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061507    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061509    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061512    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061514    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061517    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061519    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061522    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061524    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061527    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061529    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061532    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:06.073384 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061534    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061538    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061540    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061543    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061545    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061548    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061550    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061553    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061555    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061557    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061560    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061562    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061565    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061567    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061570    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061573    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061575    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061577    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061580    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:06.073867 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061582    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061586    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061589    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061592    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061594    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061598    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061600    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061603    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061605    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061608    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061610    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061613    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061615    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061618    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061620    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061623    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061625    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061628    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061631    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:06.074337 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061635    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061638    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061641    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061644    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061646    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061649    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061651    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061654    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061656    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061659    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061661    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061664    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061669    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061672    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061675    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061678    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061680    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061684    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061686    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:06.074862 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061689    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061691    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061694    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061697    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061699    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061702    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:06.061704    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.061754    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.063921    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.064973    2577 server.go:1014] "Starting client certificate rotation"
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.065067    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.065779    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.069444    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:06.075436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.072096    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:06.084179 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.084164    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:55:06.086811 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.086798    2577 log.go:25] "Validated CRI v1 image API"
Nov 02 14:55:06.087648 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.087621    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:55:06.094398 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.094379    2577 fs.go:135] Filesystem UUIDs: map[0ecad43a-670a-461b-9975-c5e7e43ce5ac:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 85f55943-baf8-4fdf-a7f8-87567dc3ab74:/dev/nvme0n1p4]
Nov 02 14:55:06.094462 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.094398    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 14:55:06.098458 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.098338    2577 manager.go:217] Machine: {Timestamp:2026-11-02 14:55:06.097894752 +0000 UTC m=+0.375760969 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119352 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d44e91a3750e817b50438266146b3 SystemUUID:ec2d44e9-1a37-50e8-17b5-0438266146b3 BootID:9507a8b1-62d3-4748-810d-ac364446f5d1 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:68:f5:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:68:f5:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:ca:8f:30:d3:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:55:06.098458 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.098449    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:55:06.098595 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.098513    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:55:06.102203 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.102176    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:55:06.102333 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.102204    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:55:06.102390 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.102335    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:55:06.102390 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.102344    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:55:06.102390 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.102353    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:06.102390 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.102377    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:06.103424 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.103414    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:55:06.107555 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.107544    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:55:06.107675 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.107666    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:55:06.108507 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.108498    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:55:06.108545 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.108510    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:55:06.108545 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.108519    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:55:06.108545 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.108528    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:55:06.108545 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.108535    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:55:06.108892 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.108879    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:06.111281 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.111258    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:55:06.119575 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.119560    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:06.120515 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120503    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:55:06.120598 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120533    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:55:06.120693 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120684    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:55:06.120722 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120699    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:55:06.120722 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120705    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:55:06.120722 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120710    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:55:06.120722 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120716    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:55:06.120722 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.120722    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:55:06.121419 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.121410    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:55:06.121449 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.121419    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:55:06.121449 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.121426    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:55:06.121449 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.121432    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:55:06.121449 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.121440    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:55:06.121449 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.121448    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:55:06.122430 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.122421    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:55:06.122462 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.122430    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:55:06.126112 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.126101    2577 server.go:1263] "Started kubelet"
Nov 02 14:55:06.126249 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.126197    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:55:06.126305 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.126273    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:55:06.126518 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.126499    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:55:06.126780 ip-10-0-133-143 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:55:06.129400 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.129358    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:55:06.129564 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.129483    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:55:06.133022 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.132998    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 14:55:06.135123 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.135102    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:06.136411 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.136395    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:55:06.136564 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.136540    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.137227 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137208    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:55:06.137227 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137226    2577 factory.go:55] Registering systemd factory
Nov 02 14:55:06.137388 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137235    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:55:06.137388 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137255    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:55:06.137388 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137259    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:55:06.137388 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137278    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:55:06.137388 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137358    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:55:06.137388 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137380    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:55:06.137607 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137480    2577 factory.go:153] Registering CRI-O factory
Nov 02 14:55:06.137607 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137491    2577 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:55:06.137607 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137505    2577 factory.go:103] Registering Raw factory
Nov 02 14:55:06.137607 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.137515    2577 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:55:06.138539 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.138524    2577 manager.go:319] Starting recovery of all containers
Nov 02 14:55:06.149407 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.149392    2577 manager.go:324] Recovery completed
Nov 02 14:55:06.151333 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.151314    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 14:55:06.154334 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.154321    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.155927 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.155901    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.156004 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.155938    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.156004 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.155953    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.156827 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.156814    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:55:06.156827 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.156821    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.156827 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.156826    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:55:06.156970 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.156841    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:55:06.157516 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.157499    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.157578 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.157530    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.157578 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.157544    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.158934 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.158923    2577 policy_none.go:50] "Start"
Nov 02 14:55:06.158976 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.158938    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:55:06.158976 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.158948    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:55:06.160578 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.160545    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 14:55:06.160666 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.160642    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 14:55:06.160906 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.160890    2577 policy_none.go:44] "Start"
Nov 02 14:55:06.161731 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.160544    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-143.ec2.internal.18e4422b7baa278c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-143.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-143.ec2.internal,},FirstTimestamp:2026-11-02 14:55:06.126075788 +0000 UTC m=+0.403942005,LastTimestamp:2026-11-02 14:55:06.126075788 +0000 UTC m=+0.403942005,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-143.ec2.internal,}"
Nov 02 14:55:06.171712 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.171695    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:06.181261 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.181183    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-143.ec2.internal.18e4422b7d719ad8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-143.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-143.ec2.internal,},FirstTimestamp:2026-11-02 14:55:06.155924184 +0000 UTC m=+0.433790401,LastTimestamp:2026-11-02 14:55:06.155924184 +0000 UTC m=+0.433790401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-143.ec2.internal,}"
Nov 02 14:55:06.181409 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.181354    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 14:55:06.181480 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.181434    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 14:55:06.181553 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.181524    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.197954    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.199577    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46vrs"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.200929    2577 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.200955    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.200963    2577 server.go:87] "Starting device plugin registration server"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.201131    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.201139    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.201241    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.201313    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.201321    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.201728    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.201772    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.203751    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-143.ec2.internal.18e4422b7d71ed41  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-143.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-143.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-143.ec2.internal,},FirstTimestamp:2026-11-02 14:55:06.155945281 +0000 UTC m=+0.433811499,LastTimestamp:2026-11-02 14:55:06.155945281 +0000 UTC m=+0.433811499,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-143.ec2.internal,}"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.208316    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.209417    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.209439    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.209469    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.209507    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 14:55:06.215245 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.211664    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46vrs"
Nov 02 14:55:06.234957 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.234936    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:06.301791 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.301750    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.302455 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.302442    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.302541 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.302466    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.302541 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.302476    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.302541 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.302495    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.310624 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.310600    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"]
Nov 02 14:55:06.310691 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.310650    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.311334 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.311319    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.311394 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.311349    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.311394 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.311375    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.313445 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.313432    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.313568 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.313553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.313603 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.313583    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.314094 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.314077    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.314176 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.314101    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.314176 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.314110    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.314176 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.314082    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.314176 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.314167    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.314317 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.314179    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.315993 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.315981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.316069 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.316002    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:06.317485 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.317466    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:06.317574 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.317494    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:06.317574 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.317503    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:06.325005 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.324900    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.325091 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.325011    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-143.ec2.internal\": node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.338644 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.338621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15b2f7b139a98a51bb6e53a9a5605774-config\") pod \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" (UID: \"15b2f7b139a98a51bb6e53a9a5605774\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.350692 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.350674    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-143.ec2.internal\" not found" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.353144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.353129    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-143.ec2.internal\" not found" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.368706 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.368683    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.439298 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.439277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15b2f7b139a98a51bb6e53a9a5605774-config\") pod \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" (UID: \"15b2f7b139a98a51bb6e53a9a5605774\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.439409 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.439346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.439409 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.439393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15b2f7b139a98a51bb6e53a9a5605774-config\") pod \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" (UID: \"15b2f7b139a98a51bb6e53a9a5605774\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.439499 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.439409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.468904 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.468888    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.540295 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.540270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.540415 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.540314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.540415 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.540278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.540415 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.540391    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.569432 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.569384    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.655700 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.655675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.658001 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:06.657985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:06.670197 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.670172    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.770273 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.770251    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.870763 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.870718    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:06.971126 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:06.971098    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.065114 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.065087    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 14:55:07.065574 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.065223    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 14:55:07.071847 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.071831    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.172052 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.171995    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.175120 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.175093    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:07.194832 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:07.194802    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b2f7b139a98a51bb6e53a9a5605774.slice/crio-7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66 WatchSource:0}: Error finding container 7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66: Status 404 returned error can't find the container with id 7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66
Nov 02 14:55:07.200991 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.200976    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:55:07.212520 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.212485    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" event={"ID":"15b2f7b139a98a51bb6e53a9a5605774","Type":"ContainerStarted","Data":"7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66"}
Nov 02 14:55:07.217041 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.217008    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:06 +0000 UTC" deadline="2028-08-06 19:43:26.005685022 +0000 UTC"
Nov 02 14:55:07.217041 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.217039    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15436h48m18.788649175s"
Nov 02 14:55:07.267459 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.267422    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:07.268628 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.268609    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:07.268707 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.268640    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:07.268707 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.268653    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:07.272824 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.272805    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.283347 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:07.283326    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041ae145e73ad7d91d16b7f00a767ff7.slice/crio-124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894 WatchSource:0}: Error finding container 124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894: Status 404 returned error can't find the container with id 124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894
Nov 02 14:55:07.374521 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.374483    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.474130 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.474086    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:07.475193 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.475174    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.576020 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.575991    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.676570 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:07.676548    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:07.764315 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.764262    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:07.837647 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.837627    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:07.861319 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.861295    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:07.862565 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.862542    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:07.891948 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:07.891841    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:08.110717 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.110658    2577 apiserver.go:52] "Watching apiserver"
Nov 02 14:55:08.136556 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.136525    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:08.145458 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.144755    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:55:08.145458 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.145130    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-j2xpj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal","openshift-multus/multus-6f4vs","openshift-multus/multus-additional-cni-plugins-72q2z","kube-system/konnectivity-agent-svzrl","openshift-multus/network-metrics-daemon-jfqf9","openshift-network-operator/iptables-alerter-jknvp","openshift-ovn-kubernetes/ovnkube-node-wjtlf","kube-system/global-pull-secret-syncer-gc6sk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f","openshift-dns/node-resolver-g4d2h","openshift-image-registry/node-ca-vm8rc","kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal","openshift-network-diagnostics/network-check-target-gnfc9"]
Nov 02 14:55:08.147936 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.147917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.148054 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.148016    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:08.150393 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.150357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:08.150498 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.150445    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:08.152832 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.152812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.155594 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.155350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.157941 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.157918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.160559 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.160540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.164487 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.164465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.167029 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.167012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.169530 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.169512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.169631 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.169576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:08.171615 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.171594    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2fmch\""
Nov 02 14:55:08.171615 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.171594    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.171772 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.171662    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:55:08.171772 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.171723    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:55:08.171772 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.171745    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.172530 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.172513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.173036 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.173013    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:55:08.173109 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.173053    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pvkf\""
Nov 02 14:55:08.173170 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.173113    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:55:08.174551 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.174524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.179328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.177525    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:55:08.179328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.177590    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:55:08.179328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.177611    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-25j7m\""
Nov 02 14:55:08.179328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.177655    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.179328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.177665    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.179658 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.179519    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.179658 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.179524    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.179658 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.179562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.179658 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.179597    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:55:08.179658 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.179633    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:55:08.181318 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.181299    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:55:08.181435 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.181357    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7sjw\""
Nov 02 14:55:08.181868 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.181852    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:55:08.182328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.182309    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:55:08.182820 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.182790    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczs\""
Nov 02 14:55:08.184378 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.184358    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.186579 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.186559    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.186671 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.186658    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62zsq\""
Nov 02 14:55:08.201317 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.201292    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:08.201482 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.201364    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7thl2"
Nov 02 14:55:08.216018 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.215996    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" event={"ID":"041ae145e73ad7d91d16b7f00a767ff7","Type":"ContainerStarted","Data":"124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894"}
Nov 02 14:55:08.238173 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.238156    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:55:08.250449 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-kubelet-config\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.250554 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcf9\" (UniqueName: \"kubernetes.io/projected/ba225e36-3f26-42cf-b829-4e3117f7e899-kube-api-access-fhcf9\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.250554 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.250554 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-socket-dir-parent\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.250554 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-systemd-units\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.250554 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx979\" (UniqueName: \"kubernetes.io/projected/f57d8416-b76f-4dbf-8fd8-3e43d001e063-kube-api-access-kx979\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fqg\" (UniqueName: \"kubernetes.io/projected/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kube-api-access-b6fqg\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-netns\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-conf-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-systemd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-env-overrides\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-system-cni-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysconfig\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.250833 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-device-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4grh\" (UniqueName: \"kubernetes.io/projected/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-kube-api-access-m4grh\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-systemd\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-sys\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-var-lib-kubelet\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.250980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-cnibin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-hostroot\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-netns\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba225e36-3f26-42cf-b829-4e3117f7e899-serviceca\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-daemon-config\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-node-log\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-netd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-socket-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rphv8\" (UniqueName: \"kubernetes.io/projected/485e5e3f-08d1-44e0-b93b-1023a530c305-kube-api-access-rphv8\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.251395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-system-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-var-lib-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-etc-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfr8\" (UniqueName: \"kubernetes.io/projected/ba12693c-0a77-49d5-a792-d420be330a7f-kube-api-access-cpfr8\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-lib-modules\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d71694a7-787c-45ec-9308-6ff029eead3f-tmp-dir\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrlp\" (UniqueName: \"kubernetes.io/projected/d71694a7-787c-45ec-9308-6ff029eead3f-kube-api-access-7xrlp\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-multus\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-binary-copy\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-modprobe-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-sys-fs\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-bin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-etc-kubernetes\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cnibin\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-conf\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.252165 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-dbus\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/485e5e3f-08d1-44e0-b93b-1023a530c305-host-slash\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6432dfb-7757-4a30-a440-8cc647505f76-konnectivity-ca\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-host\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d71694a7-787c-45ec-9308-6ff029eead3f-hosts-file\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-slash\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-log-socket\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-run\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-ovn\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-multus-certs\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.251991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-config\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba12693c-0a77-49d5-a792-d420be330a7f-ovn-node-metrics-cert\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-script-lib\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-bin\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.252960 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-kubernetes\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-tmp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8fkp\" (UniqueName: \"kubernetes.io/projected/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-kube-api-access-t8fkp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-etc-selinux\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba225e36-3f26-42cf-b829-4e3117f7e899-host\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-cni-binary-copy\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/485e5e3f-08d1-44e0-b93b-1023a530c305-iptables-alerter-script\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-kubelet\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxw2p\" (UniqueName: \"kubernetes.io/projected/9f189efb-95a0-4e40-9b7b-8076e77c5508-kube-api-access-bxw2p\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-registration-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6432dfb-7757-4a30-a440-8cc647505f76-agent-certs\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-k8s-cni-cncf-io\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-kubelet\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.253761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.252519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-tuned\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.280306 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280282    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-btkxl\""
Nov 02 14:55:08.280306 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280288    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:55:08.280465 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280348    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.280465 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280428    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.280465 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280434    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.280590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280469    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.280590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280477    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:55:08.280590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280509    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:55:08.280590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280524    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kws2w\""
Nov 02 14:55:08.280590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280547    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:55:08.280590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.280553    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x78pf\""
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-daemon-config\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-node-log\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-netd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-socket-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphv8\" (UniqueName: \"kubernetes.io/projected/485e5e3f-08d1-44e0-b93b-1023a530c305-kube-api-access-rphv8\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-system-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-var-lib-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-etc-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfr8\" (UniqueName: \"kubernetes.io/projected/ba12693c-0a77-49d5-a792-d420be330a7f-kube-api-access-cpfr8\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-lib-modules\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d71694a7-787c-45ec-9308-6ff029eead3f-tmp-dir\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrlp\" (UniqueName: \"kubernetes.io/projected/d71694a7-787c-45ec-9308-6ff029eead3f-kube-api-access-7xrlp\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-multus\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-binary-copy\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-modprobe-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.356082 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-sys-fs\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-bin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-etc-kubernetes\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cnibin\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-conf\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-dbus\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/485e5e3f-08d1-44e0-b93b-1023a530c305-host-slash\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6432dfb-7757-4a30-a440-8cc647505f76-konnectivity-ca\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.353994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-host\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d71694a7-787c-45ec-9308-6ff029eead3f-hosts-file\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-slash\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-log-socket\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-run\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-ovn\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-multus-certs\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357090 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-config\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba12693c-0a77-49d5-a792-d420be330a7f-ovn-node-metrics-cert\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-script-lib\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-bin\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-kubernetes\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-tmp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8fkp\" (UniqueName: \"kubernetes.io/projected/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-kube-api-access-t8fkp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-etc-selinux\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba225e36-3f26-42cf-b829-4e3117f7e899-host\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-cni-binary-copy\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/485e5e3f-08d1-44e0-b93b-1023a530c305-iptables-alerter-script\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-kubelet\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxw2p\" (UniqueName: \"kubernetes.io/projected/9f189efb-95a0-4e40-9b7b-8076e77c5508-kube-api-access-bxw2p\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-registration-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6432dfb-7757-4a30-a440-8cc647505f76-agent-certs\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-k8s-cni-cncf-io\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.357945 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-kubelet\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-tuned\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-kubelet-config\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcf9\" (UniqueName: \"kubernetes.io/projected/ba225e36-3f26-42cf-b829-4e3117f7e899-kube-api-access-fhcf9\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-socket-dir-parent\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354805    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-daemon-config\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-systemd-units\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx979\" (UniqueName: \"kubernetes.io/projected/f57d8416-b76f-4dbf-8fd8-3e43d001e063-kube-api-access-kx979\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fqg\" (UniqueName: \"kubernetes.io/projected/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kube-api-access-b6fqg\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354899    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-node-log\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-netns\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-conf-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354945    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-netd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.358797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.354978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-systemd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-env-overrides\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-socket-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-system-cni-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysconfig\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-device-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4grh\" (UniqueName: \"kubernetes.io/projected/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-kube-api-access-m4grh\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-systemd\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-sys\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-var-lib-kubelet\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-cnibin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-hostroot\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-netns\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba225e36-3f26-42cf-b829-4e3117f7e899-serviceca\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.359694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-etc-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355421    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355598    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-lib-modules\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355608    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-modprobe-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/485e5e3f-08d1-44e0-b93b-1023a530c305-host-slash\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355682    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-dbus\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-sys-fs\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355749    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-bin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355812    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-etc-kubernetes\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355837    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cnibin\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-conf\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-config\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-multus\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d71694a7-787c-45ec-9308-6ff029eead3f-tmp-dir\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356316    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-run\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356383    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-script-lib\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.360464 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356431    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-log-socket\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-bin\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356517    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-kubernetes\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-slash\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d71694a7-787c-45ec-9308-6ff029eead3f-hosts-file\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-ovn\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356730    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6432dfb-7757-4a30-a440-8cc647505f76-konnectivity-ca\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356790    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-multus-certs\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356843    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-binary-copy\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356862    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-system-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-sys\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.355268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-var-lib-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-var-lib-kubelet\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba225e36-3f26-42cf-b829-4e3117f7e899-host\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.356997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-netns\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysconfig\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.361103 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-cnibin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357037    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-systemd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-hostroot\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357105    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-host\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357145    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-conf-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357201    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357216    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357227    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-systemd\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.357265    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-kubelet\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.357270    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:08.361997 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.357327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret podName:453f0247-70c9-4b6c-9272-afe056657e5d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:08.857311166 +0000 UTC m=+3.135177399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret") pod "global-pull-secret-syncer-gc6sk" (UID: "453f0247-70c9-4b6c-9272-afe056657e5d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:08.371476 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371444    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-env-overrides\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.371607 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371563    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-kubelet-config\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.371675 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-socket-dir-parent\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.371675 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371661    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-system-cni-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.371769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371703    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.371769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-systemd-units\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.371954 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.371931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.372791 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.372087    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-device-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.372791 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.372194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-etc-selinux\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.372791 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.372251    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.372791 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.372582    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:08.372791 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.372665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:55:08.87263631 +0000 UTC m=+3.150502534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:08.373106 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.372819    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-netns\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.373106 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.372971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-kubelet\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.373106 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.373047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.373543 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.373437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-registration-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.373543 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.373513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-k8s-cni-cncf-io\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.374493 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.374243    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-cni-binary-copy\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.374493 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.374363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-tmp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.375371 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.375348    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6432dfb-7757-4a30-a440-8cc647505f76-agent-certs\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.375589 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.375379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba12693c-0a77-49d5-a792-d420be330a7f-ovn-node-metrics-cert\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.375663 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.375379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba225e36-3f26-42cf-b829-4e3117f7e899-serviceca\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.375854 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.375828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/485e5e3f-08d1-44e0-b93b-1023a530c305-iptables-alerter-script\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.377528 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.377483    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-tuned\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.381126 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.381105    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:08.381229 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.381149    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:08.381229 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.381166    2577 projected.go:196] Error preparing data for projected volume kube-api-access-95knt for pod openshift-network-diagnostics/network-check-target-gnfc9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:08.381299 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.381281    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt podName:4709665b-4994-4036-836f-225e49ad8233 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:08.881262571 +0000 UTC m=+3.159128799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-95knt" (UniqueName: "kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt") pod "network-check-target-gnfc9" (UID: "4709665b-4994-4036-836f-225e49ad8233") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:08.384770 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.384705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8fkp\" (UniqueName: \"kubernetes.io/projected/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-kube-api-access-t8fkp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.386773 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.386751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fqg\" (UniqueName: \"kubernetes.io/projected/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kube-api-access-b6fqg\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.386988 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.386970    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4grh\" (UniqueName: \"kubernetes.io/projected/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-kube-api-access-m4grh\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.388291 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.388248    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfr8\" (UniqueName: \"kubernetes.io/projected/ba12693c-0a77-49d5-a792-d420be330a7f-kube-api-access-cpfr8\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.389230 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.389205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphv8\" (UniqueName: \"kubernetes.io/projected/485e5e3f-08d1-44e0-b93b-1023a530c305-kube-api-access-rphv8\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.390790 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.390768    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrlp\" (UniqueName: \"kubernetes.io/projected/d71694a7-787c-45ec-9308-6ff029eead3f-kube-api-access-7xrlp\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.398751 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.398731    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx979\" (UniqueName: \"kubernetes.io/projected/f57d8416-b76f-4dbf-8fd8-3e43d001e063-kube-api-access-kx979\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.400173 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.400147    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcf9\" (UniqueName: \"kubernetes.io/projected/ba225e36-3f26-42cf-b829-4e3117f7e899-kube-api-access-fhcf9\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.400467 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.400438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxw2p\" (UniqueName: \"kubernetes.io/projected/9f189efb-95a0-4e40-9b7b-8076e77c5508-kube-api-access-bxw2p\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.468490 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.468468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:08.480055 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.480029    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f189efb_95a0_4e40_9b7b_8076e77c5508.slice/crio-3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523 WatchSource:0}: Error finding container 3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523: Status 404 returned error can't find the container with id 3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523
Nov 02 14:55:08.493491 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.493461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:08.498446 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.498426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:08.500813 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.500789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:08.503150 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.503121    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba12693c_0a77_49d5_a792_d420be330a7f.slice/crio-0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5 WatchSource:0}: Error finding container 0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5: Status 404 returned error can't find the container with id 0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5
Nov 02 14:55:08.511092 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.511054    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485e5e3f_08d1_44e0_b93b_1023a530c305.slice/crio-c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975 WatchSource:0}: Error finding container c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975: Status 404 returned error can't find the container with id c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975
Nov 02 14:55:08.514637 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.514605    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71694a7_787c_45ec_9308_6ff029eead3f.slice/crio-f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9 WatchSource:0}: Error finding container f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9: Status 404 returned error can't find the container with id f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9
Nov 02 14:55:08.523767 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.523601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:08.525779 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.525761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:08.527824 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.527807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:08.536823 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.536795    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6432dfb_7757_4a30_a440_8cc647505f76.slice/crio-6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa WatchSource:0}: Error finding container 6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa: Status 404 returned error can't find the container with id 6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa
Nov 02 14:55:08.538136 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.537989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:08.539753 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.539731    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57d8416_b76f_4dbf_8fd8_3e43d001e063.slice/crio-11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946 WatchSource:0}: Error finding container 11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946: Status 404 returned error can't find the container with id 11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946
Nov 02 14:55:08.540663 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.540600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:08.548918 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.548658    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2c0eec_9b2e_4b78_b96d_36b37dd1e2ee.slice/crio-4ab08ec64b5c34234608804cb501a9abb590e52936d9b2b359688ddfae28086f WatchSource:0}: Error finding container 4ab08ec64b5c34234608804cb501a9abb590e52936d9b2b359688ddfae28086f: Status 404 returned error can't find the container with id 4ab08ec64b5c34234608804cb501a9abb590e52936d9b2b359688ddfae28086f
Nov 02 14:55:08.552616 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:08.551923    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4870c8c3_b2d0_4cd6_b2f7_7eec96149b97.slice/crio-e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2 WatchSource:0}: Error finding container e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2: Status 404 returned error can't find the container with id e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2
Nov 02 14:55:08.673089 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.673002    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:08.858564 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.858341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:08.858564 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.858479    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:08.858564 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.858544    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret podName:453f0247-70c9-4b6c-9272-afe056657e5d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:09.858526836 +0000 UTC m=+4.136393063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret") pod "global-pull-secret-syncer-gc6sk" (UID: "453f0247-70c9-4b6c-9272-afe056657e5d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:08.959337 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.959264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:08.959486 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:08.959343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:08.959486 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.959453    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:08.959582 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.959496    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:55:09.959482463 +0000 UTC m=+4.237348683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:08.959821 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.959803    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:08.959910 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.959825    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:08.959910 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.959836    2577 projected.go:196] Error preparing data for projected volume kube-api-access-95knt for pod openshift-network-diagnostics/network-check-target-gnfc9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:08.959910 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:08.959876    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt podName:4709665b-4994-4036-836f-225e49ad8233 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:09.959861799 +0000 UTC m=+4.237728023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-95knt" (UniqueName: "kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt") pod "network-check-target-gnfc9" (UID: "4709665b-4994-4036-836f-225e49ad8233") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:09.204864 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.204752    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:08 +0000 UTC" deadline="2028-07-19 15:05:42.495020022 +0000 UTC"
Nov 02 14:55:09.204864 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.204810    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15000h10m33.290213833s"
Nov 02 14:55:09.210535 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.210306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:09.210535 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.210449    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:09.221608 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.221575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5"}
Nov 02 14:55:09.231547 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.231520    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" event={"ID":"15b2f7b139a98a51bb6e53a9a5605774","Type":"ContainerStarted","Data":"de3fa8f50d29ae7c46dc9b39362b1bf3553958f80003dce6944ed786a73a99d1"}
Nov 02 14:55:09.238545 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.238520    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f" event={"ID":"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97","Type":"ContainerStarted","Data":"e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2"}
Nov 02 14:55:09.249496 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.249468    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j2xpj" event={"ID":"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee","Type":"ContainerStarted","Data":"4ab08ec64b5c34234608804cb501a9abb590e52936d9b2b359688ddfae28086f"}
Nov 02 14:55:09.257541 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.257462    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerStarted","Data":"11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946"}
Nov 02 14:55:09.281500 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.281263    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svzrl" event={"ID":"a6432dfb-7757-4a30-a440-8cc647505f76","Type":"ContainerStarted","Data":"6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa"}
Nov 02 14:55:09.289691 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.289527    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g4d2h" event={"ID":"d71694a7-787c-45ec-9308-6ff029eead3f","Type":"ContainerStarted","Data":"f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9"}
Nov 02 14:55:09.297223 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.297147    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6f4vs" event={"ID":"9f189efb-95a0-4e40-9b7b-8076e77c5508","Type":"ContainerStarted","Data":"3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523"}
Nov 02 14:55:09.308152 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.308094    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm8rc" event={"ID":"ba225e36-3f26-42cf-b829-4e3117f7e899","Type":"ContainerStarted","Data":"3e17fd38eb1db1e69f953cdf9e052c83e1e39db3f73c631c0eef1f9a0bbeb827"}
Nov 02 14:55:09.313300 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.313053    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jknvp" event={"ID":"485e5e3f-08d1-44e0-b93b-1023a530c305","Type":"ContainerStarted","Data":"c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975"}
Nov 02 14:55:09.866694 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.866109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:09.866694 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.866261    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:09.866694 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.866321    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret podName:453f0247-70c9-4b6c-9272-afe056657e5d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:11.866301809 +0000 UTC m=+6.144168033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret") pod "global-pull-secret-syncer-gc6sk" (UID: "453f0247-70c9-4b6c-9272-afe056657e5d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:09.966703 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.966668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:09.966891 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:09.966736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:09.966891 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.966855    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:09.966891 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.966883    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:09.967042 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.966904    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:09.967042 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.966912    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:55:11.966892629 +0000 UTC m=+6.244758834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:09.967042 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.966919    2577 projected.go:196] Error preparing data for projected volume kube-api-access-95knt for pod openshift-network-diagnostics/network-check-target-gnfc9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:09.967042 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:09.966973    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt podName:4709665b-4994-4036-836f-225e49ad8233 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:11.966955354 +0000 UTC m=+6.244821564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-95knt" (UniqueName: "kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt") pod "network-check-target-gnfc9" (UID: "4709665b-4994-4036-836f-225e49ad8233") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:10.207614 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.206036    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:08 +0000 UTC" deadline="2028-08-18 14:03:38.895420044 +0000 UTC"
Nov 02 14:55:10.207614 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.206070    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15719h8m28.689353549s"
Nov 02 14:55:10.210814 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.210788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:10.210945 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:10.210919    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:10.211598 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.211450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:10.211598 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:10.211554    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:10.328455 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.328403    2577 generic.go:356] "Generic (PLEG): container finished" podID="041ae145e73ad7d91d16b7f00a767ff7" containerID="20c7b4e8d1e5af794c7935612b033a7e8ed191eb5b905cb4202f8a256df554f3" exitCode=0
Nov 02 14:55:10.329367 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.329345    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" event={"ID":"041ae145e73ad7d91d16b7f00a767ff7","Type":"ContainerDied","Data":"20c7b4e8d1e5af794c7935612b033a7e8ed191eb5b905cb4202f8a256df554f3"}
Nov 02 14:55:10.378774 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:10.378712    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" podStartSLOduration=2.378693932 podStartE2EDuration="2.378693932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:09.280713692 +0000 UTC m=+3.558579924" watchObservedRunningTime="2026-11-02 14:55:10.378693932 +0000 UTC m=+4.656560163"
Nov 02 14:55:11.210322 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:11.209820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:11.210322 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.209962    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:11.337064 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:11.336348    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" event={"ID":"041ae145e73ad7d91d16b7f00a767ff7","Type":"ContainerStarted","Data":"bfb7ec3d524e7f1c20d6502e0dc193f66506d86cf423a034e51f75540eb1719a"}
Nov 02 14:55:11.881956 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:11.881912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:11.882166 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.882083    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:11.882166 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.882149    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret podName:453f0247-70c9-4b6c-9272-afe056657e5d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:15.882131068 +0000 UTC m=+10.159997288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret") pod "global-pull-secret-syncer-gc6sk" (UID: "453f0247-70c9-4b6c-9272-afe056657e5d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:11.982787 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:11.982701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:11.982787 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:11.982778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:11.983004 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.982896    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:11.983004 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.982956    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:55:15.982938431 +0000 UTC m=+10.260804655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:11.983466 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.983424    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:11.983466 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.983469    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:11.983627 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.983483    2577 projected.go:196] Error preparing data for projected volume kube-api-access-95knt for pod openshift-network-diagnostics/network-check-target-gnfc9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:11.983627 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:11.983530    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt podName:4709665b-4994-4036-836f-225e49ad8233 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:15.983515169 +0000 UTC m=+10.261381395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-95knt" (UniqueName: "kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt") pod "network-check-target-gnfc9" (UID: "4709665b-4994-4036-836f-225e49ad8233") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:12.209751 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:12.209721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:12.209925 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:12.209827    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:12.210206 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:12.210181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:12.210334 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:12.210283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:12.349004 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:12.348926    2577 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="935128c1caab5883ad929eb7b4a353bc3d3d7a745589504a94acbe1181d600b8" exitCode=0
Nov 02 14:55:12.349975 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:12.349947    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerDied","Data":"935128c1caab5883ad929eb7b4a353bc3d3d7a745589504a94acbe1181d600b8"}
Nov 02 14:55:12.404677 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:12.404618    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" podStartSLOduration=4.404599602 podStartE2EDuration="4.404599602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:11.386350347 +0000 UTC m=+5.664216576" watchObservedRunningTime="2026-11-02 14:55:12.404599602 +0000 UTC m=+6.682465831"
Nov 02 14:55:13.210161 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:13.209644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:13.210161 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:13.209789    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:13.365611 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:13.365571    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svzrl" event={"ID":"a6432dfb-7757-4a30-a440-8cc647505f76","Type":"ContainerStarted","Data":"8e783e273f316adbe29d3ff39dd384e015ab6b0f619f1bb8c47122c061eabdcc"}
Nov 02 14:55:14.214532 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.213862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:14.214532 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:14.213971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:14.214532 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.214331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:14.214532 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:14.214441    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:14.370183 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.369898    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm8rc" event={"ID":"ba225e36-3f26-42cf-b829-4e3117f7e899","Type":"ContainerStarted","Data":"7e7ac0a4179a93facdf8d2470ec01c6db7aab38a3c7ff21be26af6739a35e459"}
Nov 02 14:55:14.413498 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.413446    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-svzrl" podStartSLOduration=3.456917713 podStartE2EDuration="7.413429511s" totalImagesPullingTime="3.956511798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:13.416926043 +0000 UTC m=+7.694792272" watchObservedRunningTime="2026-11-02 14:55:14.413429511 +0000 UTC m=+8.691295739"
Nov 02 14:55:14.556594 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.556450    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:14.557136 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.556935    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:14.603764 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:14.603422    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vm8rc" podStartSLOduration=2.508830796 podStartE2EDuration="7.603405831s" totalImagesPullingTime="5.094575035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:14.414003408 +0000 UTC m=+8.691869634" watchObservedRunningTime="2026-11-02 14:55:14.603405831 +0000 UTC m=+8.881272058"
Nov 02 14:55:15.210016 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:15.209939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:15.210184 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:15.210072    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:15.389984 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:15.389939    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f" event={"ID":"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97","Type":"ContainerStarted","Data":"1e64c61ba2156d00f1d42bac8c895e7506f6f25267b4a2ac0b3b6e095831c3ab"}
Nov 02 14:55:15.390482 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:15.390009    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:15.390822 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:15.390803    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:15.919252 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:15.918858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:15.919252 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:15.918983    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:15.919252 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:15.919041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret podName:453f0247-70c9-4b6c-9272-afe056657e5d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:23.919023093 +0000 UTC m=+18.196889317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret") pod "global-pull-secret-syncer-gc6sk" (UID: "453f0247-70c9-4b6c-9272-afe056657e5d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:16.019914 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.019878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:16.020061 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.019951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:16.020133 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.020097    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:16.020133 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.020117    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:16.020133 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.020130    2577 projected.go:196] Error preparing data for projected volume kube-api-access-95knt for pod openshift-network-diagnostics/network-check-target-gnfc9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:16.020262 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.020207    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt podName:4709665b-4994-4036-836f-225e49ad8233 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:24.020169073 +0000 UTC m=+18.298035292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-95knt" (UniqueName: "kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt") pod "network-check-target-gnfc9" (UID: "4709665b-4994-4036-836f-225e49ad8233") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:16.020662 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.020638    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:16.020768 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.020698    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:55:24.020682819 +0000 UTC m=+18.298549023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:16.080024 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.079996    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:55:16.207560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.207439    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:55:16.080014483Z","UUID":"2f37f57c-5483-4a63-b036-f4a4ad4c9438","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:55:16.211615 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.211589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:16.211728 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.211708    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:16.212052 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.212030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:16.212573 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.212496    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:55:16.212573 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.212530    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:55:16.213157 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:16.213131    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:16.394435 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.394383    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6f4vs" event={"ID":"9f189efb-95a0-4e40-9b7b-8076e77c5508","Type":"ContainerStarted","Data":"e0d3e98840abca82a305709d503b5be29e53a5f20ddf9a43fca67e80eef9b0b3"}
Nov 02 14:55:16.398079 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.397984    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f" event={"ID":"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97","Type":"ContainerStarted","Data":"fe951a375a7a69282c9063e1d2daaabf48b58dbafc101ee7575726c0c196dcbd"}
Nov 02 14:55:16.448228 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:16.448172    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6f4vs" podStartSLOduration=2.671018798 podStartE2EDuration="10.448153277s" totalImagesPullingTime="7.777134479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:16.447330089 +0000 UTC m=+10.725196316" watchObservedRunningTime="2026-11-02 14:55:16.448153277 +0000 UTC m=+10.726019504"
Nov 02 14:55:17.210318 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.210135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:17.210478 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:17.210452    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:17.402658 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.402591    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f" event={"ID":"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97","Type":"ContainerStarted","Data":"48cd333930f9ae40a974ad811b977e7b8b82c3eeb2b7fbfe1656245c053080ca"}
Nov 02 14:55:17.405397 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.405351    2577 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="b7d8ba40ae37a0cabc2cd2ac0a723cb833000dd018ff988ab15da09356684bdf" exitCode=0
Nov 02 14:55:17.405526 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.405444    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerDied","Data":"b7d8ba40ae37a0cabc2cd2ac0a723cb833000dd018ff988ab15da09356684bdf"}
Nov 02 14:55:17.410399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.410373    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g4d2h" event={"ID":"d71694a7-787c-45ec-9308-6ff029eead3f","Type":"ContainerStarted","Data":"f54596cf76d28284f5dc30c58340bc6ba6bf76a42a34a7f05e08af6b50025ade"}
Nov 02 14:55:17.449326 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.449282    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f" podStartSLOduration=2.21210201 podStartE2EDuration="10.449269992s" totalImagesPullingTime="8.237167982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:17.449016653 +0000 UTC m=+11.726882879" watchObservedRunningTime="2026-11-02 14:55:17.449269992 +0000 UTC m=+11.727136218"
Nov 02 14:55:17.543311 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:17.543252    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g4d2h" podStartSLOduration=2.573937275 podStartE2EDuration="10.543235618s" totalImagesPullingTime="7.969298343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:17.543223759 +0000 UTC m=+11.821089986" watchObservedRunningTime="2026-11-02 14:55:17.543235618 +0000 UTC m=+11.821101845"
Nov 02 14:55:18.212797 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:18.212772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:18.212935 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:18.212878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:18.213189 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:18.213170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:18.213300 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:18.213238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:18.413633 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:18.413554    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jknvp" event={"ID":"485e5e3f-08d1-44e0-b93b-1023a530c305","Type":"ContainerStarted","Data":"cbde8fb2a104c4ccaa01451cfdde7ee47e952e1227c45a698dcb466e0e6d7740"}
Nov 02 14:55:18.457392 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:18.457324    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jknvp" podStartSLOduration=4.268187905 podStartE2EDuration="12.457312211s" totalImagesPullingTime="8.189124306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:18.456956115 +0000 UTC m=+12.734822361" watchObservedRunningTime="2026-11-02 14:55:18.457312211 +0000 UTC m=+12.735178436"
Nov 02 14:55:19.209913 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:19.209851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:19.210035 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:19.209975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:19.417608 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:19.417574    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j2xpj" event={"ID":"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee","Type":"ContainerStarted","Data":"493ff1ce083223fd8170564a5421066f9591d143eb7f19958ed849810b500213"}
Nov 02 14:55:19.419995 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:19.419955    2577 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="9ca729a9ec79fe2a82e5d01db3e08b588ce43d1fe9ba45649749d6796ea5f3ad" exitCode=0
Nov 02 14:55:19.420119 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:19.420038    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerDied","Data":"9ca729a9ec79fe2a82e5d01db3e08b588ce43d1fe9ba45649749d6796ea5f3ad"}
Nov 02 14:55:19.463665 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:19.463601    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j2xpj" podStartSLOduration=2.150199267 podStartE2EDuration="12.463590219s" totalImagesPullingTime="10.313390952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:19.463171131 +0000 UTC m=+13.741037357" watchObservedRunningTime="2026-11-02 14:55:19.463590219 +0000 UTC m=+13.741456445"
Nov 02 14:55:20.212206 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:20.212180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:20.212324 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:20.212302    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:20.212653 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:20.212635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:20.212738 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:20.212720    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:20.425849 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:20.424575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerDied","Data":"a977e8e596a972adfc5f1d82cd0f89508e19d40abfcd101b10807a59d90002cd"}
Nov 02 14:55:20.425849 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:20.424560    2577 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="a977e8e596a972adfc5f1d82cd0f89508e19d40abfcd101b10807a59d90002cd" exitCode=0
Nov 02 14:55:21.209884 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:21.209850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:21.210046 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:21.209982    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:22.209918 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:22.209721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:22.210432 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:22.209765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:22.210432 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:22.210032    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:22.210432 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:22.210101    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:23.210283 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:23.210256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:23.210671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:23.210358    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:23.433491 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:23.433461    2577 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="9cc719df34329b5307d08bbd905fbc2d65e22008d99bb8656a3e90ed54794d32" exitCode=0
Nov 02 14:55:23.433575 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:23.433496    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerDied","Data":"9cc719df34329b5307d08bbd905fbc2d65e22008d99bb8656a3e90ed54794d32"}
Nov 02 14:55:23.980028 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:23.979841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:23.980028 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:23.979985    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:23.980154 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:23.980037    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret podName:453f0247-70c9-4b6c-9272-afe056657e5d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.980024462 +0000 UTC m=+34.257890680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret") pod "global-pull-secret-syncer-gc6sk" (UID: "453f0247-70c9-4b6c-9272-afe056657e5d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:24.081034 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.081011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:24.081133 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.081051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:24.081168 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.081146    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:24.081168 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.081151    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:24.081224 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.081192    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:55:40.081180007 +0000 UTC m=+34.359046210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:24.081224 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.081156    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:24.081224 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.081211    2577 projected.go:196] Error preparing data for projected volume kube-api-access-95knt for pod openshift-network-diagnostics/network-check-target-gnfc9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:24.081308 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.081232    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt podName:4709665b-4994-4036-836f-225e49ad8233 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:40.081225787 +0000 UTC m=+34.359092014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-95knt" (UniqueName: "kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt") pod "network-check-target-gnfc9" (UID: "4709665b-4994-4036-836f-225e49ad8233") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:24.212334 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.212313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:24.212644 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.212313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:24.212644 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.212420    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:24.212644 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:24.212473    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:24.438268 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.438239    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"3cd138d0c32a80ff8e4321c55f40834ca6bfa2eb2ab8691d5d08cdbece72ac35"}
Nov 02 14:55:24.438359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.438312    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"a8c8d1d6e696f0d8de8f38d48e5985afa48b20004de28c463e4548f8706892af"}
Nov 02 14:55:24.438359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.438342    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"c2bf736b0d909dc5b03a78da1aeec4d1ccd7e5664754ee218238e55ffcb5588c"}
Nov 02 14:55:24.438456 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.438388    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"b1f68486fd179e2c376e046d1af2a85e7206dca759dab6f0f2d0a631673c1ce9"}
Nov 02 14:55:24.438456 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.438417    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"f3259fe6034fb8c48314eca2c215cba182df2d5f10834beb7b96dd875a1a4707"}
Nov 02 14:55:24.438456 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.438445    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"e91d1345b266abe35366ba9053eda9fda788056f4719a8894f31b08c0a5534a8"}
Nov 02 14:55:24.441344 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.441317    2577 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="6de1c423e640c69afe50405318a29228fd6b8177cedc135c4f1512848e03fe6b" exitCode=0
Nov 02 14:55:24.441430 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:24.441357    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerDied","Data":"6de1c423e640c69afe50405318a29228fd6b8177cedc135c4f1512848e03fe6b"}
Nov 02 14:55:25.210379 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:25.210334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:25.210496 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:25.210456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:25.446198 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:25.446170    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72q2z" event={"ID":"f57d8416-b76f-4dbf-8fd8-3e43d001e063","Type":"ContainerStarted","Data":"cda80d0aec25f02dba947645e24afa8c1d95ff26886b4aedd5b0cf2ea868e08a"}
Nov 02 14:55:25.524558 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:25.524493    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-72q2z" podStartSLOduration=7.455530574 podStartE2EDuration="18.524482102s" totalImagesPullingTime="11.068951528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:25.524135543 +0000 UTC m=+19.802001770" watchObservedRunningTime="2026-11-02 14:55:25.524482102 +0000 UTC m=+19.802348327"
Nov 02 14:55:26.212553 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:26.212529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:26.212659 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:26.212529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:26.212659 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:26.212616    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:26.212719 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:26.212688    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:26.450936 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:26.450910    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"85f22d8259808c55a867edd38dbdbe1b766d3e8b1231b0d95fa40d32cf56c6ca"}
Nov 02 14:55:27.210443 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:27.210418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:27.210566 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:27.210514    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:28.210410 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.210380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:28.211033 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.210385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:28.211033 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:28.210493    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:28.211033 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:28.210583    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:28.459511 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.459484    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" event={"ID":"ba12693c-0a77-49d5-a792-d420be330a7f","Type":"ContainerStarted","Data":"138b3cd844cb83f6ad08e401f4dde9919f5a27b6260da1a99f9af1546a0c6e21"}
Nov 02 14:55:28.459791 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.459772    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:28.459855 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.459843    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:28.474454 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.474437    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:28.559129 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:28.559089    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf" podStartSLOduration=7.499547369 podStartE2EDuration="22.559077639s" totalImagesPullingTime="15.05953027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:28.558562725 +0000 UTC m=+22.836428951" watchObservedRunningTime="2026-11-02 14:55:28.559077639 +0000 UTC m=+22.836943861"
Nov 02 14:55:29.210651 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:29.210609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:29.211325 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:29.211294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:29.461459 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:29.461387    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:29.475302 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:29.475275    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:30.135043 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:30.135016    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gc6sk"]
Nov 02 14:55:30.135240 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:30.135118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:30.135240 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:30.135196    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:30.140706 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:30.140450    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gnfc9"]
Nov 02 14:55:30.140706 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:30.140562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:30.140706 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:30.140660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:30.141056 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:30.141030    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jfqf9"]
Nov 02 14:55:30.141157 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:30.141111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:30.141229 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:30.141210    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:32.210440 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:32.210409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:32.210882 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:32.210475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:32.210882 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:32.210495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:32.210882 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:32.210591    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:32.210882 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:32.210650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:32.210882 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:32.210707    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:34.210417 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:34.210390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:34.210417 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:34.210417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:34.210896 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:34.210390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:34.210896 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:34.210507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfqf9" podUID="a3be13e2-c47a-4a45-b023-fff5e1ec32ee"
Nov 02 14:55:34.210896 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:34.210558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnfc9" podUID="4709665b-4994-4036-836f-225e49ad8233"
Nov 02 14:55:34.210896 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:34.210650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc6sk" podUID="453f0247-70c9-4b6c-9272-afe056657e5d"
Nov 02 14:55:36.020876 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.020847    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeReady"
Nov 02 14:55:36.021320 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.021010    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 14:55:36.071182 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.071157    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bt2l4"]
Nov 02 14:55:36.126386 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.126346    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhlcp"]
Nov 02 14:55:36.126608 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.126455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:36.143006 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.142982    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zp66b"]
Nov 02 14:55:36.143250 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.143086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:36.146764 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.146722    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.146764 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.146727    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.163849 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.163154    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f8679cb6-t6942"]
Nov 02 14:55:36.163849 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.163275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:36.175453 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.175435    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-659675459f-cp5t4"]
Nov 02 14:55:36.175572 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.175554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.192145 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.192124    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"]
Nov 02 14:55:36.192248 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.192235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.193250 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.193230    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 14:55:36.193543 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.193522    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dmmqr\""
Nov 02 14:55:36.193639 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.193615    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 14:55:36.193639 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.193631    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 14:55:36.194521 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.194506    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8nlz6"]
Nov 02 14:55:36.194624 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.194610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.197836 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.197821    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"]
Nov 02 14:55:36.197953 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.197937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.204338 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.204312    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"]
Nov 02 14:55:36.204541 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.204518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.206800 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.206787    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"]
Nov 02 14:55:36.206934 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.206911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.208489 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.208473    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 14:55:36.208650 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.208633    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-phchw\""
Nov 02 14:55:36.209352 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.209335    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 14:55:36.209448 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.209384    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 14:55:36.210705 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.210686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:36.210705 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.210703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:36.210854 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.210726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:36.211013 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.210987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:36.214679 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.214661    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"]
Nov 02 14:55:36.219112 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.219095    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"]
Nov 02 14:55:36.219211 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.219172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.221261 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.221227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-svzrl_a6432dfb-7757-4a30-a440-8cc647505f76/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:36.235575 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.235544    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-588df7d7d7-sv5v5"]
Nov 02 14:55:36.235940 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.235923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.246114 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246096    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.246560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246451    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.246560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246487    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 14:55:36.246560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246490    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.246560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246516    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.246560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246459    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.246907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246804    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.247018 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246816    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.247018 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246847    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246846    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246854    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246863    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246882    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246883    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.246993    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.247004    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.247095    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.247121    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.247128    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.247137    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.247169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.247162    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.255212 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.255192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.255457 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.255200    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"]
Nov 02 14:55:36.262345 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.262301    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 14:55:36.272061 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n2x\" (UniqueName: \"kubernetes.io/projected/eb5081a6-d3c8-4971-9e46-49f330c7913f-kube-api-access-m4n2x\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.272061 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6831662e-48ae-42bc-95b4-27633e745504-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.272061 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdnc\" (UniqueName: \"kubernetes.io/projected/4678cafc-05c2-429b-83b6-3caa694c7d0f-kube-api-access-bwdnc\") pod \"volume-data-source-validator-58c9f5f46d-w84hz\" (UID: \"4678cafc-05c2-429b-83b6-3caa694c7d0f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:36.272228 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-default-certificate\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.272228 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcfn\" (UniqueName: \"kubernetes.io/projected/bca16489-34d0-4230-b21a-7ea110ff8c7f-kube-api-access-kpcfn\") pod \"kas-connection-checker-685895c7c9-dhlcp\" (UID: \"bca16489-34d0-4230-b21a-7ea110ff8c7f\") " pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:36.272228 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.272399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.272399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c08878-17e0-4207-8b21-a3e3da129d37-serving-cert\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.272399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-tmp\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.272399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.272566 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrwz\" (UniqueName: \"kubernetes.io/projected/f54d3276-9a6f-498e-84ff-a48b1651bc09-kube-api-access-bhrwz\") pod \"kas-connection-checker-685895c7c9-zp66b\" (UID: \"f54d3276-9a6f-498e-84ff-a48b1651bc09\") " pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:36.272566 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89m4\" (UniqueName: \"kubernetes.io/projected/60554d74-3158-455b-bfef-2d8b4e9e6feb-kube-api-access-h89m4\") pod \"kas-connection-checker-685895c7c9-bt2l4\" (UID: \"60554d74-3158-455b-bfef-2d8b4e9e6feb\") " pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:36.272566 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vhh\" (UniqueName: \"kubernetes.io/projected/e6beba97-264e-4107-bafa-57b5380ad218-kube-api-access-m4vhh\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.272566 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.272566 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6831662e-48ae-42bc-95b4-27633e745504-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.272566 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-stats-auth\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.272734 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.272734 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb5081a6-d3c8-4971-9e46-49f330c7913f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.272734 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dr6\" (UniqueName: \"kubernetes.io/projected/113ca502-9379-4a88-b3e7-3599fe9cbfd9-kube-api-access-p8dr6\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.272819 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-service-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.272819 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-config\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.272819 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.272901 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.272901 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.272959 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.272959 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.272959 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87z87\" (UniqueName: \"kubernetes.io/projected/40c08878-17e0-4207-8b21-a3e3da129d37-kube-api-access-87z87\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.273046 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113ca502-9379-4a88-b3e7-3599fe9cbfd9-serving-cert\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.273046 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.272990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.273046 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.273032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.273130 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.273056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.273130 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.273097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-snapshots\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.273130 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.273118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrk8k\" (UniqueName: \"kubernetes.io/projected/6831662e-48ae-42bc-95b4-27633e745504-kube-api-access-jrk8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.275403 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.275384    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 14:55:36.275893 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.275877    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 14:55:36.275995 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.275977    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 14:55:36.281521 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.281505    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"]
Nov 02 14:55:36.281615 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.281601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.292781 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.292763    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 14:55:36.293200 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293180    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 14:55:36.293269 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293235    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 14:55:36.293269 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293250    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 14:55:36.293269 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293238    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 14:55:36.293384 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293286    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:55:36.293666 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293649    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 14:55:36.293743 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293691    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:55:36.293743 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293701    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:55:36.293743 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293691    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-swqf5"]
Nov 02 14:55:36.293743 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:36.293864 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293749    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:55:36.293864 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.293736    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:55:36.298959 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.298943    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 14:55:36.299073 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.299043    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8vd5b\""
Nov 02 14:55:36.304057 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.304043    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bt2l4"]
Nov 02 14:55:36.304117 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.304071    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d8jgh"]
Nov 02 14:55:36.304169 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.304157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.318688 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318672    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhlcp"]
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318698    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zp66b"]
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318718    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"]
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318732    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-659675459f-cp5t4"]
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318746    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f8679cb6-t6942"]
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318759    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"]
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.318769 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318772    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318787    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8nlz6"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318803    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318817    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318828    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588df7d7d7-sv5v5"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318840    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318852    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318871    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8jgh"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318888    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"]
Nov 02 14:55:36.319026 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.318906    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-swqf5"]
Nov 02 14:55:36.346983 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.346963    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 14:55:36.347052 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.346989    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.347052 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.347005    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.347162 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.347122    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.347162 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.347139    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.347248 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.347176    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 14:55:36.347248 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.347195    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:55:36.374086 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.374162 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.374162 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6831662e-48ae-42bc-95b4-27633e745504-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.374234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdnc\" (UniqueName: \"kubernetes.io/projected/4678cafc-05c2-429b-83b6-3caa694c7d0f-kube-api-access-bwdnc\") pod \"volume-data-source-validator-58c9f5f46d-w84hz\" (UID: \"4678cafc-05c2-429b-83b6-3caa694c7d0f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:36.374280 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-default-certificate\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.374313 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcfn\" (UniqueName: \"kubernetes.io/projected/bca16489-34d0-4230-b21a-7ea110ff8c7f-kube-api-access-kpcfn\") pod \"kas-connection-checker-685895c7c9-dhlcp\" (UID: \"bca16489-34d0-4230-b21a-7ea110ff8c7f\") " pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:36.374313 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.374401 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.374401 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.374401 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fph8h\" (UniqueName: \"kubernetes.io/projected/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-kube-api-access-fph8h\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.374501 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c08878-17e0-4207-8b21-a3e3da129d37-serving-cert\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.374501 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-config-volume\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.374501 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-tmp\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.374597 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.374597 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc383de8-7d40-46ee-84ff-74b20040ff2c-config\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.374597 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrwz\" (UniqueName: \"kubernetes.io/projected/f54d3276-9a6f-498e-84ff-a48b1651bc09-kube-api-access-bhrwz\") pod \"kas-connection-checker-685895c7c9-zp66b\" (UID: \"f54d3276-9a6f-498e-84ff-a48b1651bc09\") " pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:36.374597 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89m4\" (UniqueName: \"kubernetes.io/projected/60554d74-3158-455b-bfef-2d8b4e9e6feb-kube-api-access-h89m4\") pod \"kas-connection-checker-685895c7c9-bt2l4\" (UID: \"60554d74-3158-455b-bfef-2d8b4e9e6feb\") " pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:36.374786 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vhh\" (UniqueName: \"kubernetes.io/projected/e6beba97-264e-4107-bafa-57b5380ad218-kube-api-access-m4vhh\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.374786 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.374913 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.374892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.375090 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.375069    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:36.375201 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.375201 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.375186    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.875166348 +0000 UTC m=+31.153032557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:36.375201 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.375193    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:36.375359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-tmp\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.375359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.375359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.375359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-tmp-dir\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.375359 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.375327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.875311468 +0000 UTC m=+31.153177676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:36.375643 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6831662e-48ae-42bc-95b4-27633e745504-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.375643 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc383de8-7d40-46ee-84ff-74b20040ff2c-serving-cert\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.375643 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzpw\" (UniqueName: \"kubernetes.io/projected/f523968a-ce21-493a-80f9-f02a142a320a-kube-api-access-7tzpw\") pod \"network-check-source-6f78d59f99-hv272\" (UID: \"f523968a-ce21-493a-80f9-f02a142a320a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:36.375643 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-stats-auth\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.375643 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.375892 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb5081a6-d3c8-4971-9e46-49f330c7913f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.375892 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dr6\" (UniqueName: \"kubernetes.io/projected/113ca502-9379-4a88-b3e7-3599fe9cbfd9-kube-api-access-p8dr6\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.375892 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-service-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.375892 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-config\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.375892 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.376124 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.376124 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.376124 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.375951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.376124 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6831662e-48ae-42bc-95b4-27633e745504-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.376287 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.376287 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.376358 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.376436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.376481 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.376446    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.876431509 +0000 UTC m=+31.154297734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:36.376528 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.376528 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.376625 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf141b7d-a42e-46d1-b71b-987f2c080604-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.376674 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87z87\" (UniqueName: \"kubernetes.io/projected/40c08878-17e0-4207-8b21-a3e3da129d37-kube-api-access-87z87\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.376674 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113ca502-9379-4a88-b3e7-3599fe9cbfd9-serving-cert\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.376771 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.376771 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxwm\" (UniqueName: \"kubernetes.io/projected/dc383de8-7d40-46ee-84ff-74b20040ff2c-kube-api-access-9pxwm\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.376771 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.376914 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.376914 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-snapshots\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.376914 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk8k\" (UniqueName: \"kubernetes.io/projected/6831662e-48ae-42bc-95b4-27633e745504-kube-api-access-jrk8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.376914 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.376914 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n2x\" (UniqueName: \"kubernetes.io/projected/eb5081a6-d3c8-4971-9e46-49f330c7913f-kube-api-access-m4n2x\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.377148 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.376911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78vt\" (UniqueName: \"kubernetes.io/projected/86f99765-b728-464e-bce3-fb178f34d0ed-kube-api-access-j78vt\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.377232 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.377210    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:36.377284 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.377237    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f8679cb6-t6942: secret "image-registry-tls" not found
Nov 02 14:55:36.377315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.377300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls podName:0f276b8e-7e4f-4980-a8a3-9fed6ee92685 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.877283066 +0000 UTC m=+31.155149284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls") pod "image-registry-84f8679cb6-t6942" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685") : secret "image-registry-tls" not found
Nov 02 14:55:36.387761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.387697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-default-certificate\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.387761 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.387753    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-stats-auth\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.387890 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.387777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-service-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.387962 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.387945    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.387996 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.387951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c08878-17e0-4207-8b21-a3e3da129d37-serving-cert\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.388048 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-snapshots\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.388189 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.388247 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388215    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb5081a6-d3c8-4971-9e46-49f330c7913f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.388301 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.388346 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388302    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6831662e-48ae-42bc-95b4-27633e745504-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.388447 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388424    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.388571 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.388550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-config\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.389987 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.389970    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113ca502-9379-4a88-b3e7-3599fe9cbfd9-serving-cert\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.390061 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.389997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.390182 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.390166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.393231 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.393202    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:55:36.393324 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.393237    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pj6nr\""
Nov 02 14:55:36.393324 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.393288    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:55:36.393456 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.393419    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jmg9z\""
Nov 02 14:55:36.400181 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.400154    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdnc\" (UniqueName: \"kubernetes.io/projected/4678cafc-05c2-429b-83b6-3caa694c7d0f-kube-api-access-bwdnc\") pod \"volume-data-source-validator-58c9f5f46d-w84hz\" (UID: \"4678cafc-05c2-429b-83b6-3caa694c7d0f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:36.401212 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.401107    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.401471 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.401454    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89m4\" (UniqueName: \"kubernetes.io/projected/60554d74-3158-455b-bfef-2d8b4e9e6feb-kube-api-access-h89m4\") pod \"kas-connection-checker-685895c7c9-bt2l4\" (UID: \"60554d74-3158-455b-bfef-2d8b4e9e6feb\") " pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:36.401675 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.401658    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87z87\" (UniqueName: \"kubernetes.io/projected/40c08878-17e0-4207-8b21-a3e3da129d37-kube-api-access-87z87\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.401781 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.401761    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrwz\" (UniqueName: \"kubernetes.io/projected/f54d3276-9a6f-498e-84ff-a48b1651bc09-kube-api-access-bhrwz\") pod \"kas-connection-checker-685895c7c9-zp66b\" (UID: \"f54d3276-9a6f-498e-84ff-a48b1651bc09\") " pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:36.401944 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.401930    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcfn\" (UniqueName: \"kubernetes.io/projected/bca16489-34d0-4230-b21a-7ea110ff8c7f-kube-api-access-kpcfn\") pod \"kas-connection-checker-685895c7c9-dhlcp\" (UID: \"bca16489-34d0-4230-b21a-7ea110ff8c7f\") " pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:36.403143 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.403127    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vhh\" (UniqueName: \"kubernetes.io/projected/e6beba97-264e-4107-bafa-57b5380ad218-kube-api-access-m4vhh\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.405359 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.405334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n2x\" (UniqueName: \"kubernetes.io/projected/eb5081a6-d3c8-4971-9e46-49f330c7913f-kube-api-access-m4n2x\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.406223 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.406199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dr6\" (UniqueName: \"kubernetes.io/projected/113ca502-9379-4a88-b3e7-3599fe9cbfd9-kube-api-access-p8dr6\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.406223 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.406213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk8k\" (UniqueName: \"kubernetes.io/projected/6831662e-48ae-42bc-95b4-27633e745504-kube-api-access-jrk8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.406419 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.406402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.419253 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.416735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-svzrl_a6432dfb-7757-4a30-a440-8cc647505f76/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:36.452004 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.451984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:36.471460 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.471440    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:36.477894 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.477872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc383de8-7d40-46ee-84ff-74b20040ff2c-serving-cert\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.477976 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.477904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzpw\" (UniqueName: \"kubernetes.io/projected/f523968a-ce21-493a-80f9-f02a142a320a-kube-api-access-7tzpw\") pod \"network-check-source-6f78d59f99-hv272\" (UID: \"f523968a-ce21-493a-80f9-f02a142a320a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:36.477976 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.477939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.477976 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.477968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.478136 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.477995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.478136 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.478086    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:55:36.478234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.478234 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.478172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert podName:bf141b7d-a42e-46d1-b71b-987f2c080604 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.97812688 +0000 UTC m=+31.255993104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cfnwj" (UID: "bf141b7d-a42e-46d1-b71b-987f2c080604") : secret "networking-console-plugin-cert" not found
Nov 02 14:55:36.478346 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.478346 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf141b7d-a42e-46d1-b71b-987f2c080604-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.478457 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxwm\" (UniqueName: \"kubernetes.io/projected/dc383de8-7d40-46ee-84ff-74b20040ff2c-kube-api-access-9pxwm\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.478457 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.478557 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78vt\" (UniqueName: \"kubernetes.io/projected/86f99765-b728-464e-bce3-fb178f34d0ed-kube-api-access-j78vt\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.478557 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lsp\" (UniqueName: \"kubernetes.io/projected/c9af00a4-820f-473f-bc2d-7141fbe02975-kube-api-access-z2lsp\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.478557 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.478557 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.478557 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.478552    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph8h\" (UniqueName: \"kubernetes.io/projected/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-kube-api-access-fph8h\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.478613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls podName:86f99765-b728-464e-bce3-fb178f34d0ed nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.978597917 +0000 UTC m=+31.256464142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l66th" (UID: "86f99765-b728-464e-bce3-fb178f34d0ed") : secret "samples-operator-tls" not found
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-config-volume\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc383de8-7d40-46ee-84ff-74b20040ff2c-config\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af00a4-820f-473f-bc2d-7141fbe02975-cert\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.478796 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478821    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.478847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-tmp-dir\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.479053    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf141b7d-a42e-46d1-b71b-987f2c080604-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.479144    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.479157    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588df7d7d7-sv5v5: secret "image-registry-tls" not found
Nov 02 14:55:36.479179 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.479170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.479534 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.479195    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls podName:58e33558-be4f-4b8b-812e-f73322516ac0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.979182281 +0000 UTC m=+31.257048499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls") pod "image-registry-588df7d7d7-sv5v5" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0") : secret "image-registry-tls" not found
Nov 02 14:55:36.479534 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.479272    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:55:36.479534 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.479325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls podName:8ca09ed6-20a8-4b83-bae6-a371cbf71d65 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:36.979310745 +0000 UTC m=+31.257176968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls") pod "dns-default-swqf5" (UID: "8ca09ed6-20a8-4b83-bae6-a371cbf71d65") : secret "dns-default-metrics-tls" not found
Nov 02 14:55:36.479534 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.479450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-tmp-dir\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.479789 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.479649    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-config-volume\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.479899 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.479880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc383de8-7d40-46ee-84ff-74b20040ff2c-config\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.480213 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.480193    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc383de8-7d40-46ee-84ff-74b20040ff2c-serving-cert\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.480295 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.480199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.480565 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.480549    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.480644 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.480603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.481899 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.481879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.492786 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.492769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:36.503967 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.503951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzpw\" (UniqueName: \"kubernetes.io/projected/f523968a-ce21-493a-80f9-f02a142a320a-kube-api-access-7tzpw\") pod \"network-check-source-6f78d59f99-hv272\" (UID: \"f523968a-ce21-493a-80f9-f02a142a320a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:36.504025 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.503993    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78vt\" (UniqueName: \"kubernetes.io/projected/86f99765-b728-464e-bce3-fb178f34d0ed-kube-api-access-j78vt\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.504068 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.504028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.504068 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.504045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.507059 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.507041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph8h\" (UniqueName: \"kubernetes.io/projected/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-kube-api-access-fph8h\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.507154 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.507041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxwm\" (UniqueName: \"kubernetes.io/projected/dc383de8-7d40-46ee-84ff-74b20040ff2c-kube-api-access-9pxwm\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.518554 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.518524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:36.536121 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.536096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:36.556535 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.555163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:36.582659 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.581083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af00a4-820f-473f-bc2d-7141fbe02975-cert\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.582659 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.581312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lsp\" (UniqueName: \"kubernetes.io/projected/c9af00a4-820f-473f-bc2d-7141fbe02975-kube-api-access-z2lsp\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.587538 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.587511    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af00a4-820f-473f-bc2d-7141fbe02975-cert\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.600383 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.600330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:36.613976 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.613953    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lsp\" (UniqueName: \"kubernetes.io/projected/c9af00a4-820f-473f-bc2d-7141fbe02975-kube-api-access-z2lsp\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.649500 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.649469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:36.678349 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.678318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:36.734695 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.734673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:36.837317 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.837296    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zp66b"]
Nov 02 14:55:36.842653 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.840842    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54d3276_9a6f_498e_84ff_a48b1651bc09.slice/crio-a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7 WatchSource:0}: Error finding container a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7: Status 404 returned error can't find the container with id a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7
Nov 02 14:55:36.844137 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.844028    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhlcp"]
Nov 02 14:55:36.845269 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.845220    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60554d74_3158_455b_bfef_2d8b4e9e6feb.slice/crio-02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5 WatchSource:0}: Error finding container 02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5: Status 404 returned error can't find the container with id 02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5
Nov 02 14:55:36.845339 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.845285    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bt2l4"]
Nov 02 14:55:36.846659 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.846637    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca16489_34d0_4230_b21a_7ea110ff8c7f.slice/crio-8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3 WatchSource:0}: Error finding container 8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3: Status 404 returned error can't find the container with id 8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3
Nov 02 14:55:36.884782 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.884481    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8nlz6"]
Nov 02 14:55:36.884782 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.884538    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"]
Nov 02 14:55:36.886031 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.886009    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c08878_17e0_4207_8b21_a3e3da129d37.slice/crio-afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35 WatchSource:0}: Error finding container afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35: Status 404 returned error can't find the container with id afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35
Nov 02 14:55:36.887040 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.887010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:36.887128 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.887097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.887182 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.887133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:36.887241 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.887217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887412    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.88735921 +0000 UTC m=+32.165225437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887496    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887509    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f8679cb6-t6942: secret "image-registry-tls" not found
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887545    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls podName:0f276b8e-7e4f-4980-a8a3-9fed6ee92685 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.887532463 +0000 UTC m=+32.165398684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls") pod "image-registry-84f8679cb6-t6942" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685") : secret "image-registry-tls" not found
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887609    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887642    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.88763179 +0000 UTC m=+32.165498007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887698    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:36.887894 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.887730    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.887720284 +0000 UTC m=+32.165586506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:36.912749 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.912716    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"]
Nov 02 14:55:36.916409 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.916357    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6831662e_48ae_42bc_95b4_27633e745504.slice/crio-8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7 WatchSource:0}: Error finding container 8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7: Status 404 returned error can't find the container with id 8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7
Nov 02 14:55:36.944729 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.944662    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"]
Nov 02 14:55:36.954205 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.954180    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4678cafc_05c2_429b_83b6_3caa694c7d0f.slice/crio-be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0 WatchSource:0}: Error finding container be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0: Status 404 returned error can't find the container with id be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0
Nov 02 14:55:36.971022 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.971001    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"]
Nov 02 14:55:36.972954 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:36.972932    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc383de8_7d40_46ee_84ff_74b20040ff2c.slice/crio-494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243 WatchSource:0}: Error finding container 494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243: Status 404 returned error can't find the container with id 494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243
Nov 02 14:55:36.987659 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.987636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:36.987730 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.987682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:36.987784 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.987758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:36.987821 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:36.987786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:36.987821 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987786    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:55:36.987905 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987823    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:55:36.987905 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987831    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:55:36.987905 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987851    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls podName:8ca09ed6-20a8-4b83-bae6-a371cbf71d65 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.987835534 +0000 UTC m=+32.265701758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls") pod "dns-default-swqf5" (UID: "8ca09ed6-20a8-4b83-bae6-a371cbf71d65") : secret "dns-default-metrics-tls" not found
Nov 02 14:55:36.987905 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls podName:86f99765-b728-464e-bce3-fb178f34d0ed nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.987866242 +0000 UTC m=+32.265732446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l66th" (UID: "86f99765-b728-464e-bce3-fb178f34d0ed") : secret "samples-operator-tls" not found
Nov 02 14:55:36.987905 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987883    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert podName:bf141b7d-a42e-46d1-b71b-987f2c080604 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.987877914 +0000 UTC m=+32.265744118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cfnwj" (UID: "bf141b7d-a42e-46d1-b71b-987f2c080604") : secret "networking-console-plugin-cert" not found
Nov 02 14:55:36.988122 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987938    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:36.988122 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.987956    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588df7d7d7-sv5v5: secret "image-registry-tls" not found
Nov 02 14:55:36.988122 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:36.988011    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls podName:58e33558-be4f-4b8b-812e-f73322516ac0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.98799368 +0000 UTC m=+32.265859891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls") pod "image-registry-588df7d7d7-sv5v5" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0") : secret "image-registry-tls" not found
Nov 02 14:55:37.008470 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.008433    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"]
Nov 02 14:55:37.008554 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:37.008480    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf523968a_ce21_493a_80f9_f02a142a320a.slice/crio-acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b WatchSource:0}: Error finding container acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b: Status 404 returned error can't find the container with id acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b
Nov 02 14:55:37.042905 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.042853    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8jgh"]
Nov 02 14:55:37.043521 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:37.043499    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9af00a4_820f_473f_bc2d_7141fbe02975.slice/crio-d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e WatchSource:0}: Error finding container d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e: Status 404 returned error can't find the container with id d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e
Nov 02 14:55:37.479493 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.479463    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" event={"ID":"113ca502-9379-4a88-b3e7-3599fe9cbfd9","Type":"ContainerStarted","Data":"85e4510a6b2d1e576f795074f2021e95d249c42c181dc359bfd3a09b215ccaba"}
Nov 02 14:55:37.480863 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.480838    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz" event={"ID":"4678cafc-05c2-429b-83b6-3caa694c7d0f","Type":"ContainerStarted","Data":"be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0"}
Nov 02 14:55:37.482216 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.482186    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272" event={"ID":"f523968a-ce21-493a-80f9-f02a142a320a","Type":"ContainerStarted","Data":"acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b"}
Nov 02 14:55:37.483545 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.483514    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz" event={"ID":"dc383de8-7d40-46ee-84ff-74b20040ff2c","Type":"ContainerStarted","Data":"494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243"}
Nov 02 14:55:37.484830 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.484810    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8jgh" event={"ID":"c9af00a4-820f-473f-bc2d-7141fbe02975","Type":"ContainerStarted","Data":"d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e"}
Nov 02 14:55:37.486724 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.486698    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" event={"ID":"40c08878-17e0-4207-8b21-a3e3da129d37","Type":"ContainerStarted","Data":"afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35"}
Nov 02 14:55:37.488105 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.488084    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42" event={"ID":"6831662e-48ae-42bc-95b4-27633e745504","Type":"ContainerStarted","Data":"8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7"}
Nov 02 14:55:37.489947 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.489923    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zp66b" event={"ID":"f54d3276-9a6f-498e-84ff-a48b1651bc09","Type":"ContainerStarted","Data":"99985ced656cf8617ac0d3e4ca3a7deb1b72d8f47ba4dc47de2e427af7b34556"}
Nov 02 14:55:37.490044 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.489965    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zp66b" event={"ID":"f54d3276-9a6f-498e-84ff-a48b1651bc09","Type":"ContainerStarted","Data":"a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7"}
Nov 02 14:55:37.492590 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.492567    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dhlcp" event={"ID":"bca16489-34d0-4230-b21a-7ea110ff8c7f","Type":"ContainerStarted","Data":"3ce4059a0a6a66b09af478c667c12a05570f5a24edeb49a6bb4d49378ea89c48"}
Nov 02 14:55:37.492699 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.492608    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dhlcp" event={"ID":"bca16489-34d0-4230-b21a-7ea110ff8c7f","Type":"ContainerStarted","Data":"8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3"}
Nov 02 14:55:37.495159 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.494863    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bt2l4" event={"ID":"60554d74-3158-455b-bfef-2d8b4e9e6feb","Type":"ContainerStarted","Data":"e20240cc1c9be04e0bfee465d25c2da7b540fb64bfc977cea0c079f64587023b"}
Nov 02 14:55:37.495159 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.494903    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bt2l4" event={"ID":"60554d74-3158-455b-bfef-2d8b4e9e6feb","Type":"ContainerStarted","Data":"02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5"}
Nov 02 14:55:37.534752 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.534713    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zp66b" podStartSLOduration=382.534702253 podStartE2EDuration="6m22.534702253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:37.534461553 +0000 UTC m=+31.812327780" watchObservedRunningTime="2026-11-02 14:55:37.534702253 +0000 UTC m=+31.812568470"
Nov 02 14:55:37.622119 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.622041    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dhlcp" podStartSLOduration=382.622023884 podStartE2EDuration="6m22.622023884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:37.579224899 +0000 UTC m=+31.857091124" watchObservedRunningTime="2026-11-02 14:55:37.622023884 +0000 UTC m=+31.899890113"
Nov 02 14:55:37.623097 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.623056    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bt2l4" podStartSLOduration=382.623046998 podStartE2EDuration="6m22.623046998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:37.621863934 +0000 UTC m=+31.899730164" watchObservedRunningTime="2026-11-02 14:55:37.623046998 +0000 UTC m=+31.900913204"
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.895486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.895578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895654    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.895635969 +0000 UTC m=+34.173502193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.895690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895710    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895723    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f8679cb6-t6942: secret "image-registry-tls" not found
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.895723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895763    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls podName:0f276b8e-7e4f-4980-a8a3-9fed6ee92685 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.895745254 +0000 UTC m=+34.173611471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls") pod "image-registry-84f8679cb6-t6942" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685") : secret "image-registry-tls" not found
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895819    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895854    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.895843911 +0000 UTC m=+34.173710132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895906    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:37.896144 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.895931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.895922923 +0000 UTC m=+34.173789145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.996337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.996420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.996531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:37.996572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.996716    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.996731    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588df7d7d7-sv5v5: secret "image-registry-tls" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.996784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls podName:58e33558-be4f-4b8b-812e-f73322516ac0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.996766947 +0000 UTC m=+34.274633155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls") pod "image-registry-588df7d7d7-sv5v5" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0") : secret "image-registry-tls" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.997103    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.997139    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls podName:8ca09ed6-20a8-4b83-bae6-a371cbf71d65 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.997129395 +0000 UTC m=+34.274995600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls") pod "dns-default-swqf5" (UID: "8ca09ed6-20a8-4b83-bae6-a371cbf71d65") : secret "dns-default-metrics-tls" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.997175    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.997197    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert podName:bf141b7d-a42e-46d1-b71b-987f2c080604 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.997189967 +0000 UTC m=+34.275056172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cfnwj" (UID: "bf141b7d-a42e-46d1-b71b-987f2c080604") : secret "networking-console-plugin-cert" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.997238    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:55:37.997315 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:37.997257    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls podName:86f99765-b728-464e-bce3-fb178f34d0ed nodeName:}" failed. No retries permitted until 2026-11-02 14:55:39.997251176 +0000 UTC m=+34.275117380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l66th" (UID: "86f99765-b728-464e-bce3-fb178f34d0ed") : secret "samples-operator-tls" not found
Nov 02 14:55:39.505099 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:39.504564    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz" event={"ID":"4678cafc-05c2-429b-83b6-3caa694c7d0f","Type":"ContainerStarted","Data":"bfbc13a82bf233cd8933f877cb6c85c738a0690ea540f556f56bcc8f07035241"}
Nov 02 14:55:39.550560 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:39.550504    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz" podStartSLOduration=354.652790859 podStartE2EDuration="5m56.550487372s" totalImagesPullingTime="1.897696513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:39.549059528 +0000 UTC m=+33.826925755" watchObservedRunningTime="2026-11-02 14:55:39.550487372 +0000 UTC m=+33.828353598"
Nov 02 14:55:39.915872 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:39.915839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:39.916011 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:39.915883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:39.916011 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:39.915938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:39.916011 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:39.915982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:39.916140 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916121    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:39.916177 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916142    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f8679cb6-t6942: secret "image-registry-tls" not found
Nov 02 14:55:39.916211 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916195    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls podName:0f276b8e-7e4f-4980-a8a3-9fed6ee92685 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:43.916180825 +0000 UTC m=+38.194047033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls") pod "image-registry-84f8679cb6-t6942" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685") : secret "image-registry-tls" not found
Nov 02 14:55:39.916599 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916568    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:39.916698 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916623    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:43.916609807 +0000 UTC m=+38.194476018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:39.916698 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916675    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:39.916792 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916706    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:43.916695093 +0000 UTC m=+38.194561302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:39.916792 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:39.916766    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:43.916756896 +0000 UTC m=+38.194623105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.016656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.016714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.016786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.016840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.016894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017383    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017399    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588df7d7d7-sv5v5: secret "image-registry-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017426    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017454    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls podName:58e33558-be4f-4b8b-812e-f73322516ac0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.017434874 +0000 UTC m=+38.295301097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls") pod "image-registry-588df7d7d7-sv5v5" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0") : secret "image-registry-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017486    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls podName:86f99765-b728-464e-bce3-fb178f34d0ed nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.01746946 +0000 UTC m=+38.295335683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l66th" (UID: "86f99765-b728-464e-bce3-fb178f34d0ed") : secret "samples-operator-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017547    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017579    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls podName:8ca09ed6-20a8-4b83-bae6-a371cbf71d65 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.017567802 +0000 UTC m=+38.295434028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls") pod "dns-default-swqf5" (UID: "8ca09ed6-20a8-4b83-bae6-a371cbf71d65") : secret "dns-default-metrics-tls" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017647    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:55:40.017744 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.017680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert podName:bf141b7d-a42e-46d1-b71b-987f2c080604 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.017669544 +0000 UTC m=+38.295535754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cfnwj" (UID: "bf141b7d-a42e-46d1-b71b-987f2c080604") : secret "networking-console-plugin-cert" not found
Nov 02 14:55:40.020126 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.020091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:40.117715 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.117678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:40.117861 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.117764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:40.117861 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.117815    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 14:55:40.118137 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:40.117886    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs podName:a3be13e2-c47a-4a45-b023-fff5e1ec32ee nodeName:}" failed. No retries permitted until 2026-11-02 14:56:12.117864648 +0000 UTC m=+66.395730873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs") pod "network-metrics-daemon-jfqf9" (UID: "a3be13e2-c47a-4a45-b023-fff5e1ec32ee") : secret "metrics-daemon-secret" not found
Nov 02 14:55:40.130850 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.130823    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:40.186962 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.186745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:40.224959 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.214070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:40.558787 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.554540    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gnfc9"]
Nov 02 14:55:40.624349 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:40.624282    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gc6sk"]
Nov 02 14:55:40.626986 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:40.626948    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4709665b_4994_4036_836f_225e49ad8233.slice/crio-60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94 WatchSource:0}: Error finding container 60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94: Status 404 returned error can't find the container with id 60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94
Nov 02 14:55:41.515200 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.515178    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/0.log"
Nov 02 14:55:41.515338 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.515211    2577 generic.go:356] "Generic (PLEG): container finished" podID="113ca502-9379-4a88-b3e7-3599fe9cbfd9" containerID="f9286206347d2040d12da6c1c58c1e426cd6714542b1fcffe4cf59387e143762" exitCode=255
Nov 02 14:55:41.515338 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.515269    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" event={"ID":"113ca502-9379-4a88-b3e7-3599fe9cbfd9","Type":"ContainerDied","Data":"f9286206347d2040d12da6c1c58c1e426cd6714542b1fcffe4cf59387e143762"}
Nov 02 14:55:41.515571 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.515556    2577 scope.go:122] "RemoveContainer" containerID="f9286206347d2040d12da6c1c58c1e426cd6714542b1fcffe4cf59387e143762"
Nov 02 14:55:41.518268 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.518245    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gnfc9" event={"ID":"4709665b-4994-4036-836f-225e49ad8233","Type":"ContainerStarted","Data":"60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94"}
Nov 02 14:55:41.521756 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.521735    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gc6sk" event={"ID":"453f0247-70c9-4b6c-9272-afe056657e5d","Type":"ContainerStarted","Data":"ef22a8a2fcd1e26b9f417ddf96c6a756007548860e64c3d3a4a3c0ae7e824220"}
Nov 02 14:55:41.524332 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.524282    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42" event={"ID":"6831662e-48ae-42bc-95b4-27633e745504","Type":"ContainerStarted","Data":"96b13d8a18eefa16a61cbbb32a0f3b673aefc41a53c2dac2f225228cdd0bc379"}
Nov 02 14:55:41.611181 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:41.610604    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42" podStartSLOduration=354.121860881 podStartE2EDuration="5m58.610589504s" totalImagesPullingTime="4.488728623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:41.610546719 +0000 UTC m=+35.888412944" watchObservedRunningTime="2026-11-02 14:55:41.610589504 +0000 UTC m=+35.888455769"
Nov 02 14:55:42.536420 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.536354    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz" event={"ID":"dc383de8-7d40-46ee-84ff-74b20040ff2c","Type":"ContainerStarted","Data":"e9cbca2cbe50f921880e96cddfe3f2c7d8850e30796bc91179f4c4faeba206f3"}
Nov 02 14:55:42.539200 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.539174    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8jgh" event={"ID":"c9af00a4-820f-473f-bc2d-7141fbe02975","Type":"ContainerStarted","Data":"64990904b1321ccd8d6ecd5fddf6813666bc5bbf34cb2f4c61214f4d7dc98139"}
Nov 02 14:55:42.542066 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.541672    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" event={"ID":"40c08878-17e0-4207-8b21-a3e3da129d37","Type":"ContainerStarted","Data":"e65bc4955f78d08008d5c1c358ea1b6bbdc4d9b05c0bbf8cecde741fc3c25455"}
Nov 02 14:55:42.545709 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.545645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/1.log"
Nov 02 14:55:42.547234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.546083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/0.log"
Nov 02 14:55:42.547234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.546113    2577 generic.go:356] "Generic (PLEG): container finished" podID="113ca502-9379-4a88-b3e7-3599fe9cbfd9" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b" exitCode=255
Nov 02 14:55:42.547234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.546729    2577 scope.go:122] "RemoveContainer" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b"
Nov 02 14:55:42.547234 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:42.546873    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlsqw_openshift-console-operator(113ca502-9379-4a88-b3e7-3599fe9cbfd9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" podUID="113ca502-9379-4a88-b3e7-3599fe9cbfd9"
Nov 02 14:55:42.547234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.547078    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" event={"ID":"113ca502-9379-4a88-b3e7-3599fe9cbfd9","Type":"ContainerDied","Data":"2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b"}
Nov 02 14:55:42.547234 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.547114    2577 scope.go:122] "RemoveContainer" containerID="f9286206347d2040d12da6c1c58c1e426cd6714542b1fcffe4cf59387e143762"
Nov 02 14:55:42.586516 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.586459    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz" podStartSLOduration=354.293858445 podStartE2EDuration="5m59.586441069s" totalImagesPullingTime="5.292582624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:42.584400534 +0000 UTC m=+36.862266757" watchObservedRunningTime="2026-11-02 14:55:42.586441069 +0000 UTC m=+36.864307297"
Nov 02 14:55:42.630105 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.629962    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d8jgh" podStartSLOduration=1.381479097 podStartE2EDuration="6.629942715s" totalImagesPullingTime="5.248463618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:42.62872759 +0000 UTC m=+36.906593819" watchObservedRunningTime="2026-11-02 14:55:42.629942715 +0000 UTC m=+36.907808945"
Nov 02 14:55:42.733823 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.733106    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" podStartSLOduration=354.345812517 podStartE2EDuration="5m59.733088076s" totalImagesPullingTime="5.387275559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:42.732230165 +0000 UTC m=+37.010096395" watchObservedRunningTime="2026-11-02 14:55:42.733088076 +0000 UTC m=+37.010954303"
Nov 02 14:55:42.766736 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.764373    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"]
Nov 02 14:55:42.783408 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.783359    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"]
Nov 02 14:55:42.784067 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.784040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:42.800993 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.800969    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 14:55:42.800993 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.800985    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:42.801154 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.801139    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tsm8f\""
Nov 02 14:55:42.801541 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.801519    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"]
Nov 02 14:55:42.801624 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.801558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:42.801847 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.801562    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"]
Nov 02 14:55:42.960312 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.960288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wgv\" (UniqueName: \"kubernetes.io/projected/05e6fe27-aaeb-479c-9075-7590e278d505-kube-api-access-x5wgv\") pod \"migrator-66df5d496b-kg6gs\" (UID: \"05e6fe27-aaeb-479c-9075-7590e278d505\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:42.960457 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:42.960420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzrp\" (UniqueName: \"kubernetes.io/projected/717989db-0433-4e65-95a3-6a367b4e682b-kube-api-access-hvzrp\") pod \"migrator-66df5d496b-kn6nx\" (UID: \"717989db-0433-4e65-95a3-6a367b4e682b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:43.061161 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.061122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wgv\" (UniqueName: \"kubernetes.io/projected/05e6fe27-aaeb-479c-9075-7590e278d505-kube-api-access-x5wgv\") pod \"migrator-66df5d496b-kg6gs\" (UID: \"05e6fe27-aaeb-479c-9075-7590e278d505\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:43.061320 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.061191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzrp\" (UniqueName: \"kubernetes.io/projected/717989db-0433-4e65-95a3-6a367b4e682b-kube-api-access-hvzrp\") pod \"migrator-66df5d496b-kn6nx\" (UID: \"717989db-0433-4e65-95a3-6a367b4e682b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:43.103644 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.103615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wgv\" (UniqueName: \"kubernetes.io/projected/05e6fe27-aaeb-479c-9075-7590e278d505-kube-api-access-x5wgv\") pod \"migrator-66df5d496b-kg6gs\" (UID: \"05e6fe27-aaeb-479c-9075-7590e278d505\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:43.104119 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.104098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzrp\" (UniqueName: \"kubernetes.io/projected/717989db-0433-4e65-95a3-6a367b4e682b-kube-api-access-hvzrp\") pod \"migrator-66df5d496b-kn6nx\" (UID: \"717989db-0433-4e65-95a3-6a367b4e682b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:43.114328 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.114309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:43.135536 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.135513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:43.294019 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.293986    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rzg46"]
Nov 02 14:55:43.325395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.322964    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzg46"]
Nov 02 14:55:43.325395 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.323091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.340445 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.340396    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:55:43.340571 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.340538    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-st89p\""
Nov 02 14:55:43.340678 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.340608    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:55:43.363926 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.363901    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-j6hws-tdl49"]
Nov 02 14:55:43.386341 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.386290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.389153 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.389105    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j6hws-tdl49"]
Nov 02 14:55:43.440898 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.440855    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-czhhz\""
Nov 02 14:55:43.444336 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.444306    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"]
Nov 02 14:55:43.446248 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:43.446215    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717989db_0433_4e65_95a3_6a367b4e682b.slice/crio-c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452 WatchSource:0}: Error finding container c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452: Status 404 returned error can't find the container with id c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452
Nov 02 14:55:43.466482 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.466458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5150492-44d2-426a-a833-69856c7ddf09-crio-socket\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.466584 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.466499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.466584 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.466532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqkl\" (UniqueName: \"kubernetes.io/projected/e5150492-44d2-426a-a833-69856c7ddf09-kube-api-access-gvqkl\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.466707 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.466688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.466823 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.466798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5150492-44d2-426a-a833-69856c7ddf09-data-volume\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.499116 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.499091    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"]
Nov 02 14:55:43.502257 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:43.502234    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e6fe27_aaeb_479c_9075_7590e278d505.slice/crio-bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e WatchSource:0}: Error finding container bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e: Status 404 returned error can't find the container with id bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e
Nov 02 14:55:43.551285 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.551217    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272" event={"ID":"f523968a-ce21-493a-80f9-f02a142a320a","Type":"ContainerStarted","Data":"de52889b6f9d0ce2f928a37e9f4c1cf11534bbed85c8672a3b9da0d8c2c55f7d"}
Nov 02 14:55:43.552642 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.552591    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs" event={"ID":"05e6fe27-aaeb-479c-9075-7590e278d505","Type":"ContainerStarted","Data":"bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e"}
Nov 02 14:55:43.553896 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.553871    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx" event={"ID":"717989db-0433-4e65-95a3-6a367b4e682b","Type":"ContainerStarted","Data":"c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452"}
Nov 02 14:55:43.556311 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.556292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/1.log"
Nov 02 14:55:43.556852 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.556832    2577 scope.go:122] "RemoveContainer" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b"
Nov 02 14:55:43.557038 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.557014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlsqw_openshift-console-operator(113ca502-9379-4a88-b3e7-3599fe9cbfd9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" podUID="113ca502-9379-4a88-b3e7-3599fe9cbfd9"
Nov 02 14:55:43.567742 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.567841 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5150492-44d2-426a-a833-69856c7ddf09-data-volume\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.567923 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.567923 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5150492-44d2-426a-a833-69856c7ddf09-crio-socket\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.567923 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.568073 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqkl\" (UniqueName: \"kubernetes.io/projected/e5150492-44d2-426a-a833-69856c7ddf09-kube-api-access-gvqkl\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.568073 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.567975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.568073 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.568016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.568073 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.568042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.568221 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.568181    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:43.568267 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.568223    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.068208656 +0000 UTC m=+38.346074876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:43.568546 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.568519    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5150492-44d2-426a-a833-69856c7ddf09-data-volume\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.568702 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.568683    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:43.568841 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.568739    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.06872419 +0000 UTC m=+38.346590404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:43.568841 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.568773    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5150492-44d2-426a-a833-69856c7ddf09-crio-socket\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.593766 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.593740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqkl\" (UniqueName: \"kubernetes.io/projected/e5150492-44d2-426a-a833-69856c7ddf09-kube-api-access-gvqkl\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:43.594726 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.594688    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272" podStartSLOduration=350.73937343 podStartE2EDuration="5m56.594676933s" totalImagesPullingTime="5.855303503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:43.594488191 +0000 UTC m=+37.872354416" watchObservedRunningTime="2026-11-02 14:55:43.594676933 +0000 UTC m=+37.872543159"
Nov 02 14:55:43.669623 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.669587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.670000 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.669975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.670059 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.670043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.670092 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.670078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.670583 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.670557    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.670637 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.670617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.670704 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.670685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.695951 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.695924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.721907 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.721875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:43.973051 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.972974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:43.973190 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.973057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:43.973190 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973141    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:51.973121854 +0000 UTC m=+46.250988062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:43.973307 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973191    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:43.973307 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973209    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f8679cb6-t6942: secret "image-registry-tls" not found
Nov 02 14:55:43.973307 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.973210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:43.973307 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973251    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls podName:0f276b8e-7e4f-4980-a8a3-9fed6ee92685 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:51.973239445 +0000 UTC m=+46.251105676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls") pod "image-registry-84f8679cb6-t6942" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685") : secret "image-registry-tls" not found
Nov 02 14:55:43.973307 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:43.973235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:43.973307 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973310    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:43.973589 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:51.973326359 +0000 UTC m=+46.251192599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:43.973589 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973345    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:43.973589 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:43.973396    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:51.973387373 +0000 UTC m=+46.251253577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:44.075612 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.075462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:44.076404 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.075660    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:55:44.076404 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.075732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert podName:bf141b7d-a42e-46d1-b71b-987f2c080604 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.075714151 +0000 UTC m=+46.353580373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cfnwj" (UID: "bf141b7d-a42e-46d1-b71b-987f2c080604") : secret "networking-console-plugin-cert" not found
Nov 02 14:55:44.076404 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.076275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:44.076404 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.076357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076412    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076485    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:45.076468203 +0000 UTC m=+39.354334408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.076422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076506    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076519    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588df7d7d7-sv5v5: secret "image-registry-tls" not found
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076569    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls podName:58e33558-be4f-4b8b-812e-f73322516ac0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.076554074 +0000 UTC m=+46.354420299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls") pod "image-registry-588df7d7d7-sv5v5" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0") : secret "image-registry-tls" not found
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076624    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.076630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:44.076671 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls podName:86f99765-b728-464e-bce3-fb178f34d0ed nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.076657158 +0000 UTC m=+46.354523365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l66th" (UID: "86f99765-b728-464e-bce3-fb178f34d0ed") : secret "samples-operator-tls" not found
Nov 02 14:55:44.077040 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.076700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:44.077040 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076723    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:55:44.077040 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076763    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls podName:8ca09ed6-20a8-4b83-bae6-a371cbf71d65 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.076750056 +0000 UTC m=+46.354616263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls") pod "dns-default-swqf5" (UID: "8ca09ed6-20a8-4b83-bae6-a371cbf71d65") : secret "dns-default-metrics-tls" not found
Nov 02 14:55:44.077040 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076839    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:44.077040 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:44.076883    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:45.076871369 +0000 UTC m=+39.354737575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:44.082757 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.082734    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j6hws-tdl49"]
Nov 02 14:55:44.083754 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:44.083732    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c5d512_8f1d_4fbb_86d1_0651b52b965f.slice/crio-572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a WatchSource:0}: Error finding container 572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a: Status 404 returned error can't find the container with id 572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a
Nov 02 14:55:44.236555 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.236467    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b89mz"]
Nov 02 14:55:44.262285 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.262182    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b89mz"]
Nov 02 14:55:44.262452 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.262313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.280399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.280245    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 14:55:44.280399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.280268    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 14:55:44.280399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.280327    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 14:55:44.280399 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.280403    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rrjlf\""
Nov 02 14:55:44.280677 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.280417    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 14:55:44.379504 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.379447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-key\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.379706 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.379638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwq98\" (UniqueName: \"kubernetes.io/projected/127445ba-9a8b-4291-931e-4f2f5a24a41d-kube-api-access-nwq98\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.379771 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.379739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-cabundle\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.481066 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.481026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwq98\" (UniqueName: \"kubernetes.io/projected/127445ba-9a8b-4291-931e-4f2f5a24a41d-kube-api-access-nwq98\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.481229 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.481100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-cabundle\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.481229 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.481175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-key\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.482432 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.482402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-cabundle\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.483983 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.483958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-key\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.508474 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.508335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwq98\" (UniqueName: \"kubernetes.io/projected/127445ba-9a8b-4291-931e-4f2f5a24a41d-kube-api-access-nwq98\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.562597 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.562558    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6hws-tdl49" event={"ID":"48c5d512-8f1d-4fbb-86d1-0651b52b965f","Type":"ContainerStarted","Data":"b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415"}
Nov 02 14:55:44.562759 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.562618    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6hws-tdl49" event={"ID":"48c5d512-8f1d-4fbb-86d1-0651b52b965f","Type":"ContainerStarted","Data":"572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a"}
Nov 02 14:55:44.565032 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.564970    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gnfc9" event={"ID":"4709665b-4994-4036-836f-225e49ad8233","Type":"ContainerStarted","Data":"a380cbe662597570e9b23805f16491478490075077df3c19b2fdcfbae5494f46"}
Nov 02 14:55:44.565400 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.565382    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:44.588738 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.588478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:44.607506 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.607462    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-j6hws-tdl49" podStartSLOduration=1.607447595 podStartE2EDuration="1.607447595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:44.606284682 +0000 UTC m=+38.884150912" watchObservedRunningTime="2026-11-02 14:55:44.607447595 +0000 UTC m=+38.885313820"
Nov 02 14:55:44.653682 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.652957    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gnfc9" podStartSLOduration=35.522353681 podStartE2EDuration="38.652937069s" totalImagesPullingTime="3.130583388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:44.651657884 +0000 UTC m=+38.929524128" watchObservedRunningTime="2026-11-02 14:55:44.652937069 +0000 UTC m=+38.930803298"
Nov 02 14:55:44.918586 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:44.918558    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b89mz"]
Nov 02 14:55:44.930375 ip-10-0-133-143 kubenswrapper[2577]: W1102 14:55:44.930331    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127445ba_9a8b_4291_931e_4f2f5a24a41d.slice/crio-ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a WatchSource:0}: Error finding container ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a: Status 404 returned error can't find the container with id ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a
Nov 02 14:55:45.088121 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.088087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:45.088436 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.088186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:45.088436 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:45.088250    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:45.088832 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:45.088770    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:47.088741697 +0000 UTC m=+41.366607917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:45.088832 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:45.088411    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:45.088832 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:45.088813    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:47.088806125 +0000 UTC m=+41.366672342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:45.571490 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.571402    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs" event={"ID":"05e6fe27-aaeb-479c-9075-7590e278d505","Type":"ContainerStarted","Data":"5f8e88d6015a0c5d17ac5564f810dcbf7f8772760de871e9949c6c84431f6546"}
Nov 02 14:55:45.573309 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.573271    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx" event={"ID":"717989db-0433-4e65-95a3-6a367b4e682b","Type":"ContainerStarted","Data":"26e14156f374ad3a999ebbe5ab1e3cd17d2ed54deb9bbb71c13abd9765be9336"}
Nov 02 14:55:45.577489 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.577408    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz" event={"ID":"127445ba-9a8b-4291-931e-4f2f5a24a41d","Type":"ContainerStarted","Data":"2e80d97aa691e97566851c273ca0e0c33c043d0fa9a7c26909ea297b20239c8c"}
Nov 02 14:55:45.577489 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.577457    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz" event={"ID":"127445ba-9a8b-4291-931e-4f2f5a24a41d","Type":"ContainerStarted","Data":"ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a"}
Nov 02 14:55:45.620785 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.620712    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs" podStartSLOduration=1.808769572 podStartE2EDuration="3.620687929s" totalImagesPullingTime="1.811918357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:45.619123957 +0000 UTC m=+39.896990184" watchObservedRunningTime="2026-11-02 14:55:45.620687929 +0000 UTC m=+39.898554160"
Nov 02 14:55:45.665244 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:45.665174    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx" podStartSLOduration=1.800106129 podStartE2EDuration="3.665154198s" totalImagesPullingTime="1.865048069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:45.662980632 +0000 UTC m=+39.940846866" watchObservedRunningTime="2026-11-02 14:55:45.665154198 +0000 UTC m=+39.943020427"
Nov 02 14:55:46.413382 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:46.413328    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g4d2h_d71694a7-787c-45ec-9308-6ff029eead3f/dns-node-resolver/0.log"
Nov 02 14:55:46.536656 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:46.536619    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:46.536826 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:46.536686    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:46.537076 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:46.537059    2577 scope.go:122] "RemoveContainer" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b"
Nov 02 14:55:46.537285 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:46.537264    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlsqw_openshift-console-operator(113ca502-9379-4a88-b3e7-3599fe9cbfd9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" podUID="113ca502-9379-4a88-b3e7-3599fe9cbfd9"
Nov 02 14:55:47.115478 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:47.115444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:47.115638 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:47.115564    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:47.115638 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:47.115594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:47.115638 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:47.115630    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:51.115611647 +0000 UTC m=+45.393477868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:47.115742 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:47.115716    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:47.115797 ip-10-0-133-143 kubenswrapper[2577]: E1102 14:55:47.115785    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:51.11576435 +0000 UTC m=+45.393630568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:47.589308 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:47.589276    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gc6sk" event={"ID":"453f0247-70c9-4b6c-9272-afe056657e5d","Type":"ContainerStarted","Data":"b18eb4a88d8105bbe3b582e0a7ed6b772a3263b130bb786d4d1bfd2849823a6e"}
Nov 02 14:55:47.636243 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:47.636158    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz" podStartSLOduration=3.636142833 podStartE2EDuration="3.636142833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:45.715723448 +0000 UTC m=+39.993589677" watchObservedRunningTime="2026-11-02 14:55:47.636142833 +0000 UTC m=+41.914009062"
Nov 02 14:55:47.636402 ip-10-0-133-143 kubenswrapper[2577]: I1102 14:55:47.636240    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gc6sk" podStartSLOduration=33.940567972 podStartE2EDuration="40.63623547s" totalImagesPullingTime="6.695667498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:47.63488871 +0000 UTC m=+41.912754937" watchObservedRunningTime="2026-11-02 14:55:47.63623547 +0000 UTC m=+41.914101696"
Nov 02 14:55:47.682863 ip-10-0-133-143 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 14:55:47.697723 ip-10-0-133-143 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 14:55:47.697887 ip-10-0-133-143 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 14:55:47.697933 ip-10-0-133-143 systemd[1]: kubelet.service: Consumed 2.991s CPU time.
Nov 02 14:55:47.712033 ip-10-0-133-143 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:55:47.796353 ip-10-0-133-143 kubenswrapper[5041]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:47.796353 ip-10-0-133-143 kubenswrapper[5041]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:55:47.796353 ip-10-0-133-143 kubenswrapper[5041]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:47.796353 ip-10-0-133-143 kubenswrapper[5041]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:47.798485 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798416    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.798485 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798484    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.798485 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798488    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798491    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798495    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798497    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798500    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798503    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798505    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798508    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798511    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798513    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798516    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798519    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798521    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798524    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798526    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798529    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798531    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798534    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798536    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798540    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.798585 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798548    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798551    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798554    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798557    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798560    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798562    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798565    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798568    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798571    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798574    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798576    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798579    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798582    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798584    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798587    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798590    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798593    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798595    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798599    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.799093 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798604    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798607    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798610    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798612    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798615    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798617    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798620    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798623    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798626    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798629    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798632    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798634    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798637    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798640    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798643    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798646    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798649    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798651    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798654    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.799589 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798659    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798661    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798664    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798667    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798669    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798672    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798674    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798677    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798679    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798682    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798685    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798688    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798690    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798693    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798697    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798700    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798704    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798706    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798709    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.800082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798712    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798714    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798717    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798719    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798722    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798724    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798727    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798730    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798732    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798735    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798737    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798740    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798743    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798746    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798750    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798753    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798755    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798758    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798761    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798763    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.800581 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798765    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798768    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798771    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798774    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798777    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798779    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.798782    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799552    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799561    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799565    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799568    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799571    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799573    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799576    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799579    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799582    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799585    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799587    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799590    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799593    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.801097 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799596    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799600    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799603    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799605    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799610    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799614    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799617    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799619    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799622    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799625    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799627    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799630    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799632    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799635    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799638    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799640    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799643    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799645    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799648    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.801617 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799651    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799653    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799656    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799658    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799661    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799663    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799666    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799668    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799671    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799673    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799676    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799679    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799681    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799695    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799700    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799703    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799706    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799709    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799712    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799714    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.802112 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799717    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799720    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799722    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799725    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799728    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799730    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799733    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799736    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799738    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799742    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799745    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799748    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799751    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799754    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799756    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799759    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799761    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799764    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799767    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799769    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.802635 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799772    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799775    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799777    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799780    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799789    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799793    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799795    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799798    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799801    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799804    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799807    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799809    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799812    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799815    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799817    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799820    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799822    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799825    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799828    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799830    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.803152 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799833    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799835    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799838    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799840    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799844    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799846    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799849    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799851    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799854    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799856    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799859    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799861    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799864    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.799869    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800374    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800381    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800387    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800392    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800395    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.803710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800398    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800403    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800407    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800410    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800413    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800416    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800419    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800421    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800424    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800427    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800429    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800432    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800434    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800437    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800439    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800442    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800445    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800448    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800450    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.804230 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800453    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800456    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800458    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800461    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800464    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800466    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800468    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800471    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800473    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800476    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800480    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800482    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800485    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800488    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800490    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800493    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800496    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800498    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800501    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800503    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.804729 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800506    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800509    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800511    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800514    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800516    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800519    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800522    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800524    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800527    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800530    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800533    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800535    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800539    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800541    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800544    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800546    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800549    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800551    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800554    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.805278 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800557    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800559    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800561    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800565    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800568    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800571    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800573    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800576    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800578    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800581    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800584    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800587    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800590    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800592    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800595    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800598    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800600    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800602    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800605    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800608    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.805767 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800610    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800613    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800615    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800618    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800620    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800623    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800626    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800628    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800632    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800634    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800637    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800639    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800642    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800644    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800647    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800650    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800653    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800656    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800658    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.806257 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800661    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800663    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800666    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.800669    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800769    5041 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800776    5041 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800781    5041 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800786    5041 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800791    5041 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800794    5041 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800798    5041 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800802    5041 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800806    5041 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800809    5041 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800812    5041 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800815    5041 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800818    5041 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800821    5041 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800824    5041 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800827    5041 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800830    5041 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800833    5041 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800841    5041 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:55:47.806738 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800844    5041 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800847    5041 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800850    5041 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800853    5041 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800857    5041 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800861    5041 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800865    5041 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800868    5041 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800871    5041 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800874    5041 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800877    5041 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800881    5041 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800884    5041 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800891    5041 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800894    5041 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800897    5041 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800900    5041 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800903    5041 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800906    5041 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800910    5041 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800913    5041 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800916    5041 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800919    5041 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800922    5041 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:55:47.807346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800926    5041 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800929    5041 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800932    5041 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800935    5041 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800938    5041 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800941    5041 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800944    5041 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800947    5041 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800950    5041 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800953    5041 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800956    5041 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800962    5041 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800965    5041 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800970    5041 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800973    5041 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800976    5041 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800979    5041 flags.go:64] FLAG: --help="false"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800982    5041 flags.go:64] FLAG: --hostname-override="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800985    5041 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800989    5041 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800992    5041 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.800997    5041 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801001    5041 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801005    5041 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:55:47.807998 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801007    5041 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801010    5041 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801013    5041 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801016    5041 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801020    5041 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801023    5041 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801026    5041 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801028    5041 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801032    5041 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801035    5041 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801038    5041 flags.go:64] FLAG: --lock-file=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801041    5041 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801043    5041 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801046    5041 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801052    5041 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801055    5041 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801058    5041 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801061    5041 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801064    5041 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801067    5041 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801070    5041 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801075    5041 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801079    5041 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801082    5041 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801086    5041 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:55:47.808610 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801089    5041 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801092    5041 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801095    5041 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801098    5041 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801103    5041 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801106    5041 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801110    5041 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801117    5041 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801121    5041 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801124    5041 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801127    5041 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801130    5041 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801132    5041 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801136    5041 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801139    5041 flags.go:64] FLAG: --port="10250"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801142    5041 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801145    5041 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03964f7dc41bb8368"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801148    5041 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801151    5041 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801154    5041 flags.go:64] FLAG: --register-node="true"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801157    5041 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801161    5041 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801164    5041 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801167    5041 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801170    5041 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801174    5041 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:55:47.809636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801177    5041 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801180    5041 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801184    5041 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801187    5041 flags.go:64] FLAG: --runonce="false"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801190    5041 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801193    5041 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801196    5041 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801199    5041 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801202    5041 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801205    5041 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801209    5041 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801213    5041 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801216    5041 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801219    5041 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801222    5041 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801225    5041 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801228    5041 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801231    5041 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801234    5041 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801239    5041 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801243    5041 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801248    5041 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801254    5041 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801258    5041 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801263    5041 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:55:47.810319 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801267    5041 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801272    5041 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801275    5041 flags.go:64] FLAG: --v="2"
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801280    5041 flags.go:64] FLAG: --version="false"
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801284    5041 flags.go:64] FLAG: --vmodule=""
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801288    5041 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801291    5041 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801447    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801452    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801456    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801459    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801462    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801465    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801467    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801470    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801473    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801475    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801479    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801482    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801485    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801487    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801490    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.810989 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801493    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801496    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801499    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801501    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801504    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801507    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801509    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801512    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801515    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801519    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801523    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801526    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801529    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801531    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801534    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801536    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801539    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801542    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801544    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.811710 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801548    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801551    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801554    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801556    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801559    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801561    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801564    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801567    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801571    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801573    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801576    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801578    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801581    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801584    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801587    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801589    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801592    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801594    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801597    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.812260 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801600    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801602    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801605    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801607    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801610    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801613    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801615    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801618    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801620    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801623    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801626    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801629    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801631    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801637    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801640    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801643    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801645    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801648    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801650    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801653    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.813103 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801656    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801659    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801662    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801665    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801667    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801670    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801673    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801675    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801678    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801681    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801683    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801686    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801689    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801691    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801694    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801697    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801699    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801702    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801704    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.813960 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801707    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801710    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801712    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801715    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801717    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801719    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801723    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801726    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801728    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801731    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801733    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801737    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801742    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801744    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.801772    5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801805    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.814548 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801809    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801811    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801814    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801817    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801819    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801823    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801825    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801828    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801831    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801833    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801836    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801839    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801841    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801844    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801847    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801849    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801852    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801855    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801857    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801860    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.815033 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801862    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801865    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801867    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801873    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801876    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801879    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801882    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801885    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801888    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801892    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801898    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801903    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801907    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801911    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801916    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801919    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801922    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801925    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801929    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.815572 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801931    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801934    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801937    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801940    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801942    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801945    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801947    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801950    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801953    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801955    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801958    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801960    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801963    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801966    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801968    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801971    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801975    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801977    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801980    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.816054 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801982    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801985    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801988    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801990    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801994    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.801998    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802001    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802003    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802006    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802008    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802011    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802014    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802016    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802019    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802022    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802024    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802027    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802030    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802032    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.816544 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802035    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802037    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802040    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802043    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802045    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802048    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802050    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802053    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802056    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802058    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802062    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802065    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802067    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802070    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802073    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802075    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802078    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802082    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802085    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.817031 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802087    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802090    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802093    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802095    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802098    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802101    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802105    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802109    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.802112    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.805552    5041 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.805566    5041 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.805584    5041 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.805588    5041 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805646    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805651    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805655    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805658    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805660    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805663    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.817606 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805666    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805669    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805671    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805674    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805676    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805679    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805682    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805684    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805687    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805689    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805692    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805695    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805698    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805700    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805703    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805706    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805708    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805711    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805714    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.818055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805717    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805720    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805723    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805727    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805732    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805735    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805739    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805741    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805744    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805747    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805750    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805753    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805756    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805758    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805761    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805764    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805766    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805769    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805772    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.818532 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805775    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805777    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805780    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805782    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805785    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805788    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805791    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805794    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805796    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805799    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805802    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805804    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805807    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805810    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805812    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805816    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805821    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805824    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805827    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.819021 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805830    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805833    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805836    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805839    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805841    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805844    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805846    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805849    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805852    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805854    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805856    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805859    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805862    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805864    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805867    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805869    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805872    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805874    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805877    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805880    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.819524 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805882    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805885    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805888    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805890    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805893    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805895    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805898    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805900    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805903    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805905    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805909    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805911    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805914    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805916    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805920    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805922    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805924    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805927    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805929    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805932    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.820074 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805934    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805937    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.805939    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.805965    5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806089    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806093    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806096    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806099    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806101    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806104    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806106    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806109    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806111    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806114    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806116    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806119    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.820679 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806121    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806124    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806126    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806129    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806132    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806134    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806137    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806140    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806142    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806145    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806148    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806150    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806153    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806155    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806158    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806160    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806163    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806165    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806168    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806171    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.821079 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806173    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806176    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806178    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806181    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806183    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806186    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806188    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806191    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806193    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806197    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806200    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806203    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806206    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806208    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806211    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806213    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806216    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806218    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806221    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.821665 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806224    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806227    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806230    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806233    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806235    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806238    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806240    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806243    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806245    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806248    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806250    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806253    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806256    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806258    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806261    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806264    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806266    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806269    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806272    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806274    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.822136 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806277    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806279    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806282    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806284    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806286    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806289    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806292    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806294    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806297    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806300    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806302    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806305    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806308    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806310    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806313    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806316    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806319    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806321    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806324    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806326    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.822696 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806329    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806331    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806334    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806336    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806339    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806341    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806344    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806346    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806349    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806351    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806354    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806358    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806376    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806380    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806383    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.823238 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.806406    5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806438    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806442    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806445    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806449    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806453    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806456    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806459    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806462    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806464    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806467    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806470    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806473    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806475    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806478    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806481    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806483    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806486    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806488    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:47.823610 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806491    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806493    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806496    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806499    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806501    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806504    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806506    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806512    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806514    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806517    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806520    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806522    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806525    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806527    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806530    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806532    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806535    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806537    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806540    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806542    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806545    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:47.824080 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806548    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806550    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806553    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806555    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806558    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806561    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806563    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806565    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806568    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806571    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806574    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806577    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806579    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806582    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806584    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806587    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806589    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806594    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806598    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:47.824634 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806601    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806603    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806606    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806609    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806611    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806614    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806618    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806621    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806624    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806627    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806629    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806632    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806635    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806637    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806640    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806642    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806645    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806648    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806650    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:47.825135 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806653    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806656    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806659    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806661    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806664    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806666    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806669    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806672    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806674    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806677    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806679    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806683    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806687    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806690    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806692    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806695    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806697    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806700    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806703    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806706    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:47.825615 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806708    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806711    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806715    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806717    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806720    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806722    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806725    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806727    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:47.806730    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.806777    5041 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.807719    5041 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.807763    5041 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.807937    5041 server.go:1014] "Starting client certificate rotation"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.808070    5041 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.808118    5041 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:06 +0000 UTC" deadline="2028-08-11 03:29:33.956089148 +0000 UTC"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.808153    5041 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15540h33m46.147938121s"
Nov 02 14:55:47.826268 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.808351    5041 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:47.827068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.810667    5041 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:47.827068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.811833    5041 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:55:47.827068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.813873    5041 log.go:25] "Validated CRI v1 image API"
Nov 02 14:55:47.827068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.814557    5041 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:55:47.827068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.816472    5041 fs.go:135] Filesystem UUIDs: map[0ecad43a-670a-461b-9975-c5e7e43ce5ac:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 85f55943-baf8-4fdf-a7f8-87567dc3ab74:/dev/nvme0n1p4]
Nov 02 14:55:47.827357 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.816494    5041 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e17fd38eb1db1e69f953cdf9e052c83e1e39db3f73c631c0eef1f9a0bbeb827/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e17fd38eb1db1e69f953cdf9e052c83e1e39db3f73c631c0eef1f9a0bbeb827/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85e4510a6b2d1e576f795074f2021e95d249c42c181dc359bfd3a09b215ccaba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85e4510a6b2d1e576f795074f2021e95d249c42c181dc359bfd3a09b215ccaba/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef22a8a2fcd1e26b9f417ddf96c6a756007548860e64c3d3a4a3c0ae7e824220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef22a8a2fcd1e26b9f417ddf96c6a756007548860e64c3d3a4a3c0ae7e824220/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e6fe27-aaeb-479c-9075-7590e278d505/volumes/kubernetes.io~projected/kube-api-access-x5wgv:{mountpoint:/var/lib/kubelet/pods/05e6fe27-aaeb-479c-9075-7590e278d505/volumes/kubernetes.io~projected/kube-api-access-x5wgv major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~projected/kube-api-access-g464c:{mountpoint:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~projected/kube-api-access-g464c major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/113ca502-9379-4a88-b3e7-3599fe9cbfd9/volumes/kubernetes.io~projected/kube-api-access-p8dr6:{mountpoint:/var/lib/kubelet/pods/113ca502-9379-4a88-b3e7-3599fe9cbfd9/volumes/kubernetes.io~projected/kube-api-access-p8dr6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/113ca502-9379-4a88-b3e7-3599fe9cbfd9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/113ca502-9379-4a88-b3e7-3599fe9cbfd9/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/127445ba-9a8b-4291-931e-4f2f5a24a41d/volumes/kubernetes.io~projected/kube-api-access-nwq98:{mountpoint:/var/lib/kubelet/pods/127445ba-9a8b-4291-931e-4f2f5a24a41d/volumes/kubernetes.io~projected/kube-api-access-nwq98 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/127445ba-9a8b-4291-931e-4f2f5a24a41d/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/127445ba-9a8b-4291-931e-4f2f5a24a41d/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c08878-17e0-4207-8b21-a3e3da129d37/volumes/kubernetes.io~projected/kube-api-access-87z87:{mountpoint:/var/lib/kubelet/pods/40c08878-17e0-4207-8b21-a3e3da129d37/volumes/kubernetes.io~projected/kube-api-access-87z87 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c08878-17e0-4207-8b21-a3e3da129d37/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/40c08878-17e0-4207-8b21-a3e3da129d37/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453f0247-70c9-4b6c-9272-afe056657e5d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/453f0247-70c9-4b6c-9272-afe056657e5d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4678cafc-05c2-429b-83b6-3caa694c7d0f/volumes/kubernetes.io~projected/kube-api-access-bwdnc:{mountpoint:/var/lib/kubelet/pods/4678cafc-05c2-429b-83b6-3caa694c7d0f/volumes/kubernetes.io~projected/kube-api-access-bwdnc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4709665b-4994-4036-836f-225e49ad8233/volumes/kubernetes.io~projected/kube-api-access-95knt:{mountpoint:/var/lib/kubelet/pods/4709665b-4994-4036-836f-225e49ad8233/volumes/kubernetes.io~projected/kube-api-access-95knt major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/485e5e3f-08d1-44e0-b93b-1023a530c305/volumes/kubernetes.io~projected/kube-api-access-rphv8:{mountpoint:/var/lib/kubelet/pods/485e5e3f-08d1-44e0-b93b-1023a530c305/volumes/kubernetes.io~projected/kube-api-access-rphv8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97/volumes/kubernetes.io~projected/kube-api-access-b6fqg:{mountpoint:/var/lib/kubelet/pods/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97/volumes/kubernetes.io~projected/kube-api-access-b6fqg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48c5d512-8f1d-4fbb-86d1-0651b52b965f/volumes/kubernetes.io~projected/kube-api-access-wgrpn:{mountpoint:/var/lib/kubelet/pods/48c5d512-8f1d-4fbb-86d1-0651b52b965f/volumes/kubernetes.io~projected/kube-api-access-wgrpn major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~projected/kube-api-access-rzf95:{mountpoint:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~projected/kube-api-access-rzf95 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60554d74-3158-455b-bfef-2d8b4e9e6feb/volumes/kubernetes.io~projected/kube-api-access-h89m4:{mountpoint:/var/lib/kubelet/pods/60554d74-3158-455b-bfef-2d8b4e9e6feb/volumes/kubernetes.io~projected/kube-api-access-h89m4 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6831662e-48ae-42bc-95b4-27633e745504/volumes/kubernetes.io~projected/kube-api-access-jrk8k:{mountpoint:/var/lib/kubelet/pods/6831662e-48ae-42bc-95b4-27633e745504/volumes/kubernetes.io~projected/kube-api-access-jrk8k major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6831662e-48ae-42bc-95b4-27633e745504/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6831662e-48ae-42bc-95b4-27633e745504/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/717989db-0433-4e65-95a3-6a367b4e682b/volumes/kubernetes.io~projected/kube-api-access-hvzrp:{mountpoint:/var/lib/kubelet/pods/717989db-0433-4e65-95a3-6a367b4e682b/volumes/kubernetes.io~projected/kube-api-access-hvzrp major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86f99765-b728-464e-bce3-fb178f34d0ed/volumes/kubernetes.io~projected/kube-api-access-j78vt:{mountpoint:/var/lib/kubelet/pods/86f99765-b728-464e-bce3-fb178f34d0ed/volumes/kubernetes.io~projected/kube-api-access-j78vt major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ca09ed6-20a8-4b83-bae6-a371cbf71d65/volumes/kubernetes.io~projected/kube-api-access-fph8h:{mountpoint:/var/lib/kubelet/pods/8ca09ed6-20a8-4b83-bae6-a371cbf71d65/volumes/kubernetes.io~projected/kube-api-access-fph8h major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f189efb-95a0-4e40-9b7b-8076e77c5508/volumes/kubernetes.io~projected/kube-api-access-bxw2p:{mountpoint:/var/lib/kubelet/pods/9f189efb-95a0-4e40-9b7b-8076e77c5508/volumes/kubernetes.io~projected/kube-api-access-bxw2p major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3be13e2-c47a-4a45-b023-fff5e1ec32ee/volumes/kubernetes.io~projected/kube-api-access-m4grh:{mountpoint:/var/lib/kubelet/pods/a3be13e2-c47a-4a45-b023-fff5e1ec32ee/volumes/kubernetes.io~projected/kube-api-access-m4grh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6432dfb-7757-4a30-a440-8cc647505f76/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a6432dfb-7757-4a30-a440-8cc647505f76/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~projected/kube-api-access-t8fkp:{mountpoint:/var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~projected/kube-api-access-t8fkp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volumes/kubernetes.io~projected/kube-api-access-cpfr8:{mountpoint:/var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volumes/kubernetes.io~projected/kube-api-access-cpfr8 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba225e36-3f26-42cf-b829-4e3117f7e899/volumes/kubernetes.io~projected/kube-api-access-fhcf9:{mountpoint:/var/lib/kubelet/pods/ba225e36-3f26-42cf-b829-4e3117f7e899/volumes/kubernetes.io~projected/kube-api-access-fhcf9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bca16489-34d0-4230-b21a-7ea110ff8c7f/volumes/kubernetes.io~projected/kube-api-access-kpcfn:{mountpoint:/var/lib/kubelet/pods/bca16489-34d0-4230-b21a-7ea110ff8c7f/volumes/kubernetes.io~projected/kube-api-access-kpcfn major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9af00a4-820f-473f-bc2d-7141fbe02975/volumes/kubernetes.io~projected/kube-api-access-z2lsp:{mountpoint:/var/lib/kubelet/pods/c9af00a4-820f-473f-bc2d-7141fbe02975/volumes/kubernetes.io~projected/kube-api-access-z2lsp major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9af00a4-820f-473f-bc2d-7141fbe02975/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c9af00a4-820f-473f-bc2d-7141fbe02975/volumes/kubernetes.io~secret/cert major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d71694a7-787c-45ec-9308-6ff029eead3f/volumes/kubernetes.io~projected/kube-api-access-7xrlp:{mountpoint:/var/lib/kubelet/pods/d71694a7-787c-45ec-9308-6ff029eead3f/volumes/kubernetes.io~projected/kube-api-access-7xrlp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc383de8-7d40-46ee-84ff-74b20040ff2c/volumes/kubernetes.io~projected/kube-api-access-9pxwm:{mountpoint:/var/lib/kubelet/pods/dc383de8-7d40-46ee-84ff-74b20040ff2c/volumes/kubernetes.io~projected/kube-api-access-9pxwm major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc383de8-7d40-46ee-84ff-74b20040ff2c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dc383de8-7d40-46ee-84ff-74b20040ff2c/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5150492-44d2-426a-a833-69856c7ddf09/volumes/kubernetes.io~projected/kube-api-access-gvqkl:{mountpoint:/var/lib/kubelet/pods/e5150492-44d2-426a-a833-69856c7ddf09/volumes/kubernetes.io~projected/kube-api-access-gvqkl major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~projected/kube-api-access-m4vhh:{mountpoint:/var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~projected/kube-api-access-m4vhh major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb5081a6-d3c8-4971-9e46-49f330c7913f/volumes/kubernetes.io~projected/kube-api-access-m4n2x:{mountpoint:/var/lib/kubelet/pods/eb5081a6-d3c8-4971-9e46-49f330c7913f/volumes/kubernetes.io~projected/kube-api-access-m4n2x major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f523968a-ce21-493a-80f9-f02a142a320a/volumes/kubernetes.io~projected/kube-api-access-7tzpw:{mountpoint:/var/lib/kubelet/pods/f523968a-ce21-493a-80f9-f02a142a320a/volumes/kubernetes.io~projected/kube-api-access-7tzpw major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f54d3276-9a6f-498e-84ff-a48b1651bc09/volumes/kubernetes.io~projected/kube-api-access-bhrwz:{mountpoint:/var/lib/kubelet/pods/f54d3276-9a6f-498e-84ff-a48b1651bc09/volumes/kubernetes.io~projected/kube-api-access-bhrwz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f57d8416-b76f-4dbf-8fd8-3e43d001e063/volumes/kubernetes.io~projected/kube-api-access-kx979:{mountpoint:/var/lib/kubelet/pods/f57d8416-b76f-4dbf-8fd8-3e43d001e063/volumes/kubernetes.io~projected/kube-api-access-kx979 major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ad94ed9e7a5312fd87fa27634f6a1610bfe5e588f60debf267b6e3435fdef116/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/63b4d2f844344a0844715bccea880f7092e8994873de5f0bee306b247228e470/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/84befb9849714895da44e8220750f3a4b75036d25a68c08bc64c03f5c19a5b50/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9644a50b15e66ba053bb9037b6ac292b0613f9ea10d94448aa0994f40ad22b27/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/47e25cf1d9f8d633b99be3d32cb90b8078b1139bae2d9937e4718d0c74027ffa/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/010e033fe6f69c51febb5a88880bb857bb1fdcde0b58494b5d64caff543ec1da/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/682a7ac8179ba9fe7a5e8025374c0699ace115343f10a65350cead35d46923a3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/17eb7f87c1ad2671f8c7247d369b1be75e714029ea9d72406fde98c18ddd73db/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/710fa39250ce4352e235e45838c27cfc65c2a9d23572aae4ff9b547bab127584/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0f2d3f9e8b0a16bce712e3540e5e7a77f609640f0f9148f5411504faf91a95d7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/17bfffd5285bcad4b0735102119fa6b2093ce362c135d67fcec23ccf081d5456/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/27d265889d80930fa8c0cd861115513c01b00128b79eb41eff7c917c46114e07/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2f7be1a5f522b39cee337a5d888f3cbfb76c2604ed356b4e8e61147df11b32c5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/41d02d35681e023cb1464109c26547a5bd6254f468de7deb0dd03a168c1c4893/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ad23302a7884e4ef635151ecdf64ad9f1e12880113094917d2e2346531f2bcaa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9ac3be1e1a4154dca35ba382e30241e053abd31a6834e4c2c6aad341198898e5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/93b450f1629a057e8e1961ea6992178b3af09ca892ca233e1e45cb53e12434a9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c647adc1f336380b78440b88ce4fa8fc5aca15a8103c4de9e02b4d5807cd4c6a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8b729196d94e9c1843bca558bd22141b15f0d2dd2f6ef69710d234537c06d5b7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ef7a7c71a256265f5dcb25897097136769fddcf8aca5530fe0e57de8b2e73d93/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3b6efbad949bbd81f82445491217fefb097245babde9f8afd08e103bd3b210a9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/fa1a2019c625d4d24a20665829abbcaefe12e39dd7bb3839c450d23724158384/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/33e2ce9a8e7eb72204e5a15c1f7ee1dd3744d4caa0f4619d5789ee593e0aadd7/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/94a6f10e2486dcc2f78106af40e3da632e7e95873a11edc5843e4c8046e3f8e9/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/74ce1267ef7250d3262fc434957138d42868daab14aa36aab101ead37cce045a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5bc6cb158af391a96d453f3d196d035d09259656736ececc14111eee8474cdae/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6c128d40042102c5963eeb81f05167bbf62105103c999978110549191c2ab719/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b3372e90322603274386947233ee956c96c84a089bf09d49c814a70c823c1504/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f82465d0de8e2b87677094146d2de2c71bfc212b09d45a8d111ec5646757d249/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/15fce6294a819a5da6d7fa15d490d4e0ac02fa5b28ff2bb93a08d5ef00aadabb/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/cb1e3331d126198af191fdb1b763dd746bd6fdddf13dc7eca1f9678d3dc8debe/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/03c9a2f2e68273fa4c3fdb22017fc4a7affe0653a483ba80bab063f631ee8c4d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/13ac44a4bb345f243cb2a5dae3b50a0634fe4958d40581708c34f6a49d2cd500/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/2c166692cc5865b8cb0ad64048d99f4ed260f169a2da5ada1298d438d4043f15/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/888e3cdf5b6eb1c2dd87235bc3ecdece27fa2578f5fcee39fcc68e71060f6bad/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1c790ae3486de9251073774a01e0fe6c9d0aa1594d4730e27533d35c29d842b8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/0b97287917964446c42fa94d4e195a7cbcf89f2f4caee3d26025429b666c5e59/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/054fa98dc28caec5a711bc386e9b2d6cbd542c3b0f8f5a20096e2a370855c4c9/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/c502953c475100a076ac49d6a9685a5ba8b4064ebbf4446d9cf7537e3247fea3/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/e4baf1a385411df6e2202f91428027119e092d62dac063651fc92123ae419b88/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/e0bf203214706144efd5c471bda5f09808192bab00d9247e3ce32799d28d4834/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/c05320a583b3cd900cac4d79997c1d72f2f44969cf9b661124c7da1d0ac8bb60/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/5da5910eb68ba68c1337ea255a2742046743c8d6c165b5fdd9ac7ea11dd1ce7f/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/0480d7c06dc3f9703f755bcd612d37d6d0478f5bc0193f46ab22c87c4966f7b0/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c244d733c1713077d47b4f902b42cf6e8870d9f563cb814aa200bbde3cf80e1d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/f9ec3251323f614cdbb6f3321fd912e54f0e891484044471f933d349dbcdc859/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/cd0eed2e767d3eb74fca9660850335777a3fdc59a39c02fecd1856008b8d4d33/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/e2c067a3259b0b122b3c4d15cb8ae221ff281d7ab4f562f3724f286101f31b8f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/6e47b5f4a0ced6ac7a724b304bb15eb43c127b8986b6e2ae451ade72bdb7312a/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/47ac39f949641a9a61e93f3c663e9fa76b2c4252df530908e8a2886dd3ab9061/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/5cd4fb37a823703ebd0fbecfead54160a02da14c9d9bbc3c424be8a4acbacbf2/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/8d6b16a1f37a1019474f1681afe04ba1f7af9ae890e05d0d1ff2e819bc4c32be/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/7a78959ef3d69d03e6b46a6e45c8f3d97a424945a9495f25a2520289c750ba95/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/080fbb5fcdfb27597adba8eaa98367c3f3ff9f029b09d841978bff87d0d1ddca/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/28c9d053411ed0e489e20fc2d543413c4fee527b1c5a05861172f4c4775a036f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/cd1b4eb93da84eabf1289d129388969458adb646ca81bfa4676c4b916ce1a3d9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/98def3475a557c9e2c3b4a504e361151ed280b985846c6502d7c2586e1d9d467/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/899156007d49787b1305e10982c5e09e893038b0bfc403d4945eb28286a12baa/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/694e4b5e1d5e6ebb806c2ed6b12dbe0c6513c06007904618049eedecf3148b4a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f53493331d6ffbd691ed8717e55c78355500e1483463b610bdcfe00b65eda43b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8ccd845138c770d7c18af14ce9a6ea379d2f2f15ddf8126a3d6f8d9f131beba8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0727f86409619b7f115f867aa43bc889d9260eae2ba7a14bf4a4ca129d070ed8/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.824621    5041 manager.go:217] Machine: {Timestamp:2026-11-02 14:55:47.824150442 +0000 UTC m=+0.082219883 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099573 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d44e91a3750e817b50438266146b3 SystemUUID:ec2d44e9-1a37-50e8-17b5-0438266146b3 BootID:9507a8b1-62d3-4748-810d-ac364446f5d1 Filesystems:[{Device:/run/containers/storage/overlay-containers/3526e409f8be178d0cfa15717f46c9ba7feb5246c6040fea694df32e84c06523/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02f2faad476fc23cd14aadf9ffbaacc07017424ffb1625b902a4369057f065a5/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453f0247-70c9-4b6c-9272-afe056657e5d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d71694a7-787c-45ec-9308-6ff029eead3f/volumes/kubernetes.io~projected/kube-api-access-7xrlp DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef22a8a2fcd1e26b9f417ddf96c6a756007548860e64c3d3a4a3c0ae7e824220/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05e6fe27-aaeb-479c-9075-7590e278d505/volumes/kubernetes.io~projected/kube-api-access-x5wgv DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8170de91ef0aafb835f51edce139dc08c230d74ffe0e429f1a82a067e81bd8a7/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/494bc821abde6a9fc0bd08c6273ae99dcacf8b61b53c9379c888c90a108c5243/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3be13e2-c47a-4a45-b023-fff5e1ec32ee/volumes/kubernetes.io~projected/kube-api-access-m4grh DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ca09ed6-20a8-4b83-bae6-a371cbf71d65/volumes/kubernetes.io~projected/kube-api-access-fph8h DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6432dfb-7757-4a30-a440-8cc647505f76/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11e6a0b91dbe721b2e2b3db416518d47a583dc239bf53943b979ea426114e946/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e61883bad3ff7a15923b46838993d59c99c333934985a8f388be26d1e7d893d2/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8de14e75b387d7de00fbe31d33b76bcb403e5368ce97c6c3adefa65ec8890fa3/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4fdd1da129582bfaba715b696bd476345f5fe3135c0392b205418af9831e4b7/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e5a9d178dd7729e2a6a602b8d1709e073a08111285d1bfb4b3f0cd87fd75da5/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4678cafc-05c2-429b-83b6-3caa694c7d0f/volumes/kubernetes.io~projected/kube-api-access-bwdnc DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/717989db-0433-4e65-95a3-6a367b4e682b/volumes/kubernetes.io~projected/kube-api-access-hvzrp DeviceMajor:0 DeviceMinor:333 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/127445ba-9a8b-4291-931e-4f2f5a24a41d/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60554d74-3158-455b-bfef-2d8b4e9e6feb/volumes/kubernetes.io~projected/kube-api-access-h89m4 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f54d3276-9a6f-498e-84ff-a48b1651bc09/volumes/kubernetes.io~projected/kube-api-access-bhrwz DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60291bb19631027b4b5e5d52af548c1c17585ba6e2d1d8c65cc8f48387e56b94/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6831662e-48ae-42bc-95b4-27633e745504/volumes/kubernetes.io~projected/kube-api-access-jrk8k DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc383de8-7d40-46ee-84ff-74b20040ff2c/volumes/kubernetes.io~projected/kube-api-access-9pxwm DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc383de8-7d40-46ee-84ff-74b20040ff2c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e17fd38eb1db1e69f953cdf9e052c83e1e39db3f73c631c0eef1f9a0bbeb827/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9af00a4-820f-473f-bc2d-7141fbe02975/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb22e0876e8871903934d0027d96a0448cea181ad926fcd2636e82f31f06ef8e/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48c5d512-8f1d-4fbb-86d1-0651b52b965f/volumes/kubernetes.io~projected/kube-api-access-wgrpn DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/127445ba-9a8b-4291-931e-4f2f5a24a41d/volumes/kubernetes.io~projected/kube-api-access-nwq98 DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e6beba97-264e-4107-bafa-57b5380ad218/volumes/kubernetes.io~projected/kube-api-access-m4vhh DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f523968a-ce21-493a-80f9-f02a142a320a/volumes/kubernetes.io~projected/kube-api-access-7tzpw DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85e4510a6b2d1e576f795074f2021e95d249c42c181dc359bfd3a09b215ccaba/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3e224d0a744dc20565908e1a7b95db2957192252414e115424fc4ca82ebf975/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afa7193e34eaa966cc04cd5b958e81e30dad60e30a6abc9c155cdcb22f63ca35/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed26ccdaa6edc6871881ff60edb25c6f00a48a8b421b7154c7bd3972b204ae7a/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40c08878-17e0-4207-8b21-a3e3da129d37/volumes/kubernetes.io~projected/kube-api-access-87z87 DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~projected/kube-api-access-t8fkp DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb5081a6-d3c8-4971-9e46-49f330c7913f/volumes/kubernetes.io~projected/kube-api-access-m4n2x DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c343869e618976e4325948e866cf0aae0be08a30cec3e1bbfe92bb80ba350452/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5150492-44d2-426a-a833-69856c7ddf09/volumes/kubernetes.io~projected/kube-api-access-gvqkl DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f189efb-95a0-4e40-9b7b-8076e77c5508/volumes/kubernetes.io~projected/kube-api-access-bxw2p DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fde4ac15e338722da32aacbc609620857aec85cb3b5305653e752ec697196aa/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~projected/kube-api-access-g464c DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86f99765-b728-464e-bce3-fb178f34d0ed/volumes/kubernetes.io~projected/kube-api-access-j78vt DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97/volumes/kubernetes.io~projected/kube-api-access-b6fqg DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/485e5e3f-08d1-44e0-b93b-1023a530c305/volumes/kubernetes.io~projected/kube-api-access-rphv8 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba225e36-3f26-42cf-b829-4e3117f7e899/volumes/kubernetes.io~projected/kube-api-access-fhcf9 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/113ca502-9379-4a88-b3e7-3599fe9cbfd9/volumes/kubernetes.io~projected/kube-api-access-p8dr6 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be57408c9c2fa1b5179a1558e21e4bb07bc5f29516de93d9ef2129e04f3f62c0/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4709665b-4994-4036-836f-225e49ad8233/volumes/kubernetes.io~projected/kube-api-access-95knt DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f57d8416-b76f-4dbf-8fd8-3e43d001e063/volumes/kubernetes.io~projected/kube-api-access-kx979 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/40c08878-17e0-4207-8b21-a3e3da129d37/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/113ca502-9379-4a88-b3e7-3599fe9cbfd9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bca16489-34d0-4230-b21a-7ea110ff8c7f/volumes/kubernetes.io~projected/kube-api-access-kpcfn DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba12693c-0a77-49d5-a792-d420be330a7f/volumes/kubernetes.io~projected/kube-api-access-cpfr8 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6831662e-48ae-42bc-95b4-27633e745504/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes/kubernetes.io~projected/kube-api-access-rzf95 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f33f77bc2257656c3a880bfe202ef6be78aa7e31f22ce8b22b4a1918b6c8cdd9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acc1a99aac1fa45f747592c3588997101f868ed0f4c17e436f9cfd06858f2b0b/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9c30c94ec46b87862b8f199f8d48d828266b3afe51cc6d41fec4604ef49775e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9af00a4-820f-473f-bc2d-7141fbe02975/volumes/kubernetes.io~projected/kube-api-access-z2lsp DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02f2faad476fc23 MacAddress:a2:a0:e6:b7:9c:3a Speed:10000 Mtu:8901} {Name:494bc821abde6a9 MacAddress:1e:95:4d:2a:43:b8 Speed:10000 Mtu:8901} {Name:572bcbb3f960193 MacAddress:a6:61:20:99:2f:63 Speed:10000 Mtu:8901} {Name:60291bb19631027 MacAddress:ba:d8:4c:b6:01:d4 Speed:10000 Mtu:8901} {Name:8170de91ef0aafb MacAddress:56:5b:f5:e8:15:4b Speed:10000 Mtu:8901} {Name:85e4510a6b2d1e5 MacAddress:76:43:dd:b1:59:78 Speed:10000 Mtu:8901} {Name:8de14e75b387d7d MacAddress:d6:7d:f4:3c:8a:b8 Speed:10000 Mtu:8901} {Name:a4fdd1da129582b MacAddress:92:f5:83:fe:ad:59 Speed:10000 Mtu:8901} {Name:acc1a99aac1fa45 MacAddress:de:43:f3:d6:ce:9c Speed:10000 Mtu:8901} {Name:afa7193e34eaa96 MacAddress:52:82:22:e0:ac:1a Speed:10000 Mtu:8901} {Name:bb22e0876e88719 MacAddress:7e:e5:16:f0:86:84 Speed:10000 Mtu:8901} {Name:be57408c9c2fa1b MacAddress:a6:73:ae:c6:f9:00 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:68:f5:39 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:67:10:81:7b:a2 Speed:0 Mtu:8901} {Name:c343869e618976e MacAddress:e2:60:52:fe:9f:78 Speed:10000 Mtu:8901} {Name:d9c30c94ec46b87 MacAddress:aa:a1:0f:7a:e8:e0 Speed:10000 Mtu:8901} {Name:ed26ccdaa6edc68 MacAddress:a6:ef:3b:c2:e5:15 Speed:10000 Mtu:8901} {Name:ef22a8a2fcd1e26 MacAddress:7e:7d:03:64:69:55 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:68:f5:39 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:ca:8f:30:d3:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.824983    5041 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825025    5041 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825331    5041 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825349    5041 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825486    5041 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825494    5041 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825504    5041 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825521    5041 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825540    5041 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825697    5041 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825814    5041 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825847    5041 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825856    5041 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825866    5041 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825875    5041 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:55:47.827768 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.825883    5041 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:55:47.828992 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.828975    5041 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:55:47.828992 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.828992    5041 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:47.829691 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.829669    5041 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:55:47.831880 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.831862    5041 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:55:47.831880 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.831875    5041 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:47.832026 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.831973    5041 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 14:55:47.832159 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832146    5041 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:55:47.832213 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832175    5041 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:55:47.832283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832272    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832288    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832294    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832299    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832307    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832315    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832324    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832330    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:55:47.832333 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832337    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:55:47.832747 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832343    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:55:47.832747 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832351    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:55:47.832747 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832373    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:55:47.832747 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832397    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:55:47.832747 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.832407    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:55:47.835812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.835793    5041 server.go:1263] "Started kubelet"
Nov 02 14:55:47.835906 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.835863    5041 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:55:47.836083 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.836009    5041 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:55:47.836175 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.836087    5041 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:55:47.837046 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.836532    5041 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:55:47.836349 ip-10-0-133-143 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:55:47.838869 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.838854    5041 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:55:47.839445 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839428    5041 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:47.839593 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839559    5041 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:08 +0000 UTC" deadline="2028-03-31 14:27:59.702950359 +0000 UTC"
Nov 02 14:55:47.839680 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839595    5041 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12359h32m11.863358997s"
Nov 02 14:55:47.839680 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839600    5041 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:55:47.839680 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839612    5041 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:55:47.839680 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839604    5041 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:55:47.839680 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.839632    5041 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:55:47.840135 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:47.839574    5041 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-143.ec2.internal\" not found"
Nov 02 14:55:47.844715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844691    5041 factory.go:55] Registering systemd factory
Nov 02 14:55:47.844715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844717    5041 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:55:47.844901 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844852    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9af00a4-820f-473f-bc2d-7141fbe02975" volumeName="kubernetes.io/secret/c9af00a4-820f-473f-bc2d-7141fbe02975-cert" seLinuxMountContext=""
Nov 02 14:55:47.844956 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844910    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets" seLinuxMountContext=""
Nov 02 14:55:47.844956 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844930    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c08878-17e0-4207-8b21-a3e3da129d37" volumeName="kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-service-ca-bundle" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844956    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844969    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6831662e-48ae-42bc-95b4-27633e745504" volumeName="kubernetes.io/secret/6831662e-48ae-42bc-95b4-27633e745504-serving-cert" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.844989    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee" volumeName="kubernetes.io/projected/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-kube-api-access-t8fkp" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845002    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba12693c-0a77-49d5-a792-d420be330a7f" volumeName="kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-env-overrides" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845013    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71694a7-787c-45ec-9308-6ff029eead3f" volumeName="kubernetes.io/projected/d71694a7-787c-45ec-9308-6ff029eead3f-kube-api-access-7xrlp" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845039    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca09ed6-20a8-4b83-bae6-a371cbf71d65" volumeName="kubernetes.io/empty-dir/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-tmp-dir" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845054    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f189efb-95a0-4e40-9b7b-8076e77c5508" volumeName="kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845071    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f189efb-95a0-4e40-9b7b-8076e77c5508" volumeName="kubernetes.io/projected/9f189efb-95a0-4e40-9b7b-8076e77c5508-kube-api-access-bxw2p" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845087    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3be13e2-c47a-4a45-b023-fff5e1ec32ee" volumeName="kubernetes.io/projected/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-kube-api-access-m4grh" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845108    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="113ca502-9379-4a88-b3e7-3599fe9cbfd9" volumeName="kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-config" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845121    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845143    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60554d74-3158-455b-bfef-2d8b4e9e6feb" volumeName="kubernetes.io/projected/60554d74-3158-455b-bfef-2d8b4e9e6feb-kube-api-access-h89m4" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845157    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845174    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee" volumeName="kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-tmp" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845188    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845202    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48c5d512-8f1d-4fbb-86d1-0651b52b965f" volumeName="kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path" seLinuxMountContext=""
Nov 02 14:55:47.845219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845218    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86f99765-b728-464e-bce3-fb178f34d0ed" volumeName="kubernetes.io/projected/86f99765-b728-464e-bce3-fb178f34d0ed-kube-api-access-j78vt" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845231    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc383de8-7d40-46ee-84ff-74b20040ff2c" volumeName="kubernetes.io/configmap/dc383de8-7d40-46ee-84ff-74b20040ff2c-config" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845248    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6beba97-264e-4107-bafa-57b5380ad218" volumeName="kubernetes.io/projected/e6beba97-264e-4107-bafa-57b5380ad218-kube-api-access-m4vhh" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845262    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57d8416-b76f-4dbf-8fd8-3e43d001e063" volumeName="kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845276    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c08878-17e0-4207-8b21-a3e3da129d37" volumeName="kubernetes.io/projected/40c08878-17e0-4207-8b21-a3e3da129d37-kube-api-access-87z87" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845298    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4709665b-4994-4036-836f-225e49ad8233" volumeName="kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845310    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845330    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845346    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee" volumeName="kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-tuned" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845392    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845407    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717989db-0433-4e65-95a3-6a367b4e682b" volumeName="kubernetes.io/projected/717989db-0433-4e65-95a3-6a367b4e682b-kube-api-access-hvzrp" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845427    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca09ed6-20a8-4b83-bae6-a371cbf71d65" volumeName="kubernetes.io/configmap/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-config-volume" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845440    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5150492-44d2-426a-a833-69856c7ddf09" volumeName="kubernetes.io/projected/e5150492-44d2-426a-a833-69856c7ddf09-kube-api-access-gvqkl" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845454    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5081a6-d3c8-4971-9e46-49f330c7913f" volumeName="kubernetes.io/configmap/eb5081a6-d3c8-4971-9e46-49f330c7913f-telemetry-config" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845473    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54d3276-9a6f-498e-84ff-a48b1651bc09" volumeName="kubernetes.io/projected/f54d3276-9a6f-498e-84ff-a48b1651bc09-kube-api-access-bhrwz" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845485    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e6fe27-aaeb-479c-9075-7590e278d505" volumeName="kubernetes.io/projected/05e6fe27-aaeb-479c-9075-7590e278d505-kube-api-access-x5wgv" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845503    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="485e5e3f-08d1-44e0-b93b-1023a530c305" volumeName="kubernetes.io/projected/485e5e3f-08d1-44e0-b93b-1023a530c305-kube-api-access-rphv8" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845516    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845528    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc383de8-7d40-46ee-84ff-74b20040ff2c" volumeName="kubernetes.io/secret/dc383de8-7d40-46ee-84ff-74b20040ff2c-serving-cert" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845545    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c08878-17e0-4207-8b21-a3e3da129d37" volumeName="kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-tmp" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845557    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4870c8c3-b2d0-4cd6-b2f7-7eec96149b97" volumeName="kubernetes.io/projected/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kube-api-access-b6fqg" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845574    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba225e36-3f26-42cf-b829-4e3117f7e899" volumeName="kubernetes.io/projected/ba225e36-3f26-42cf-b829-4e3117f7e899-kube-api-access-fhcf9" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845587    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf141b7d-a42e-46d1-b71b-987f2c080604" volumeName="kubernetes.io/configmap/bf141b7d-a42e-46d1-b71b-987f2c080604-nginx-conf" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845603    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5081a6-d3c8-4971-9e46-49f330c7913f" volumeName="kubernetes.io/projected/eb5081a6-d3c8-4971-9e46-49f330c7913f-kube-api-access-m4n2x" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845622    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca09ed6-20a8-4b83-bae6-a371cbf71d65" volumeName="kubernetes.io/projected/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-kube-api-access-fph8h" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845634    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6432dfb-7757-4a30-a440-8cc647505f76" volumeName="kubernetes.io/secret/a6432dfb-7757-4a30-a440-8cc647505f76-agent-certs" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845653    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845665    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="113ca502-9379-4a88-b3e7-3599fe9cbfd9" volumeName="kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-trusted-ca" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845680    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e33558-be4f-4b8b-812e-f73322516ac0" volumeName="kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845700    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86f99765-b728-464e-bce3-fb178f34d0ed" volumeName="kubernetes.io/configmap/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-config" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845713    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba12693c-0a77-49d5-a792-d420be330a7f" volumeName="kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845731    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57d8416-b76f-4dbf-8fd8-3e43d001e063" volumeName="kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845743    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845756    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845778    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="113ca502-9379-4a88-b3e7-3599fe9cbfd9" volumeName="kubernetes.io/projected/113ca502-9379-4a88-b3e7-3599fe9cbfd9-kube-api-access-p8dr6" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845797    5041 factory.go:153] Registering CRI-O factory
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845801    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453f0247-70c9-4b6c-9272-afe056657e5d" volumeName="kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845812    5041 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845821    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6831662e-48ae-42bc-95b4-27633e745504" volumeName="kubernetes.io/configmap/6831662e-48ae-42bc-95b4-27633e745504-config" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845858    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6beba97-264e-4107-bafa-57b5380ad218" volumeName="kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-default-certificate" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845869    5041 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845872    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f523968a-ce21-493a-80f9-f02a142a320a" volumeName="kubernetes.io/projected/f523968a-ce21-493a-80f9-f02a142a320a-kube-api-access-7tzpw" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845888    5041 factory.go:103] Registering Raw factory
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845888    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57d8416-b76f-4dbf-8fd8-3e43d001e063" volumeName="kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 14:55:47.845996 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845904    5041 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846492    5041 manager.go:319] Starting recovery of all containers
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.845902    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="113ca502-9379-4a88-b3e7-3599fe9cbfd9" volumeName="kubernetes.io/secret/113ca502-9379-4a88-b3e7-3599fe9cbfd9-serving-cert" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846721    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="127445ba-9a8b-4291-931e-4f2f5a24a41d" volumeName="kubernetes.io/secret/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-key" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846746    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c08878-17e0-4207-8b21-a3e3da129d37" volumeName="kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-snapshots" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846765    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48c5d512-8f1d-4fbb-86d1-0651b52b965f" volumeName="kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846789    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6831662e-48ae-42bc-95b4-27633e745504" volumeName="kubernetes.io/projected/6831662e-48ae-42bc-95b4-27633e745504-kube-api-access-jrk8k" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846801    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6432dfb-7757-4a30-a440-8cc647505f76" volumeName="kubernetes.io/configmap/a6432dfb-7757-4a30-a440-8cc647505f76-konnectivity-ca" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846820    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba225e36-3f26-42cf-b829-4e3117f7e899" volumeName="kubernetes.io/configmap/ba225e36-3f26-42cf-b829-4e3117f7e899-serviceca" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846831    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9af00a4-820f-473f-bc2d-7141fbe02975" volumeName="kubernetes.io/projected/c9af00a4-820f-473f-bc2d-7141fbe02975-kube-api-access-z2lsp" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846854    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f189efb-95a0-4e40-9b7b-8076e77c5508" volumeName="kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-daemon-config" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846866    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71694a7-787c-45ec-9308-6ff029eead3f" volumeName="kubernetes.io/empty-dir/d71694a7-787c-45ec-9308-6ff029eead3f-tmp-dir" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846881    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc383de8-7d40-46ee-84ff-74b20040ff2c" volumeName="kubernetes.io/projected/dc383de8-7d40-46ee-84ff-74b20040ff2c-kube-api-access-9pxwm" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846894    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48c5d512-8f1d-4fbb-86d1-0651b52b965f" volumeName="kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846910    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba12693c-0a77-49d5-a792-d420be330a7f" volumeName="kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-config" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846943    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba12693c-0a77-49d5-a792-d420be330a7f" volumeName="kubernetes.io/secret/ba12693c-0a77-49d5-a792-d420be330a7f-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846966    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48c5d512-8f1d-4fbb-86d1-0651b52b965f" volumeName="kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846981    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c08878-17e0-4207-8b21-a3e3da129d37" volumeName="kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.846997    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4678cafc-05c2-429b-83b6-3caa694c7d0f" volumeName="kubernetes.io/projected/4678cafc-05c2-429b-83b6-3caa694c7d0f-kube-api-access-bwdnc" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847008    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bca16489-34d0-4230-b21a-7ea110ff8c7f" volumeName="kubernetes.io/projected/bca16489-34d0-4230-b21a-7ea110ff8c7f-kube-api-access-kpcfn" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847028    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5150492-44d2-426a-a833-69856c7ddf09" volumeName="kubernetes.io/empty-dir/e5150492-44d2-426a-a833-69856c7ddf09-data-volume" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847039    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57d8416-b76f-4dbf-8fd8-3e43d001e063" volumeName="kubernetes.io/projected/f57d8416-b76f-4dbf-8fd8-3e43d001e063-kube-api-access-kx979" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847056    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" volumeName="kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847067    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="127445ba-9a8b-4291-931e-4f2f5a24a41d" volumeName="kubernetes.io/projected/127445ba-9a8b-4291-931e-4f2f5a24a41d-kube-api-access-nwq98" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847080    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c08878-17e0-4207-8b21-a3e3da129d37" volumeName="kubernetes.io/secret/40c08878-17e0-4207-8b21-a3e3da129d37-serving-cert" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847096    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba12693c-0a77-49d5-a792-d420be330a7f" volumeName="kubernetes.io/projected/ba12693c-0a77-49d5-a792-d420be330a7f-kube-api-access-cpfr8" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847109    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6beba97-264e-4107-bafa-57b5380ad218" volumeName="kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-stats-auth" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847124    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="127445ba-9a8b-4291-931e-4f2f5a24a41d" volumeName="kubernetes.io/configmap/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-cabundle" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847136    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="485e5e3f-08d1-44e0-b93b-1023a530c305" volumeName="kubernetes.io/configmap/485e5e3f-08d1-44e0-b93b-1023a530c305-iptables-alerter-script" seLinuxMountContext=""
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847147    5041 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:55:47.847929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.847155    5041 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:55:47.861992 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.860480    5041 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:55:47.867809 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.867748    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/1.log"
Nov 02 14:55:47.868817 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.868796    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/1.log"
Nov 02 14:55:47.879054 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.879028    5041 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:55:47.879054 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.879056    5041 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:55:47.879210 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.879079    5041 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:55:47.879210 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:47.879116    5041 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 14:55:47.895438 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.892124    5041 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 14:55:47.895438 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.892209    5041 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:47.895438 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.892749    5041 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:47.897882 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.897859    5041 generic.go:356] "Generic (PLEG): container finished" podID="041ae145e73ad7d91d16b7f00a767ff7" containerID="20c7b4e8d1e5af794c7935612b033a7e8ed191eb5b905cb4202f8a256df554f3" exitCode=0
Nov 02 14:55:47.904149 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.904131    5041 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:47.904490 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.904470    5041 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:47.909954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.908253    5041 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="6de1c423e640c69afe50405318a29228fd6b8177cedc135c4f1512848e03fe6b" exitCode=0
Nov 02 14:55:47.909954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.908277    5041 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="9cc719df34329b5307d08bbd905fbc2d65e22008d99bb8656a3e90ed54794d32" exitCode=0
Nov 02 14:55:47.909954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.908293    5041 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="a977e8e596a972adfc5f1d82cd0f89508e19d40abfcd101b10807a59d90002cd" exitCode=0
Nov 02 14:55:47.909954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.908303    5041 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="9ca729a9ec79fe2a82e5d01db3e08b588ce43d1fe9ba45649749d6796ea5f3ad" exitCode=0
Nov 02 14:55:47.909954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.908311    5041 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="b7d8ba40ae37a0cabc2cd2ac0a723cb833000dd018ff988ab15da09356684bdf" exitCode=0
Nov 02 14:55:47.909954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.908322    5041 generic.go:356] "Generic (PLEG): container finished" podID="f57d8416-b76f-4dbf-8fd8-3e43d001e063" containerID="935128c1caab5883ad929eb7b4a353bc3d3d7a745589504a94acbe1181d600b8" exitCode=0
Nov 02 14:55:47.917919 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.917897    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/1.log"
Nov 02 14:55:47.917988 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.917936    5041 generic.go:356] "Generic (PLEG): container finished" podID="113ca502-9379-4a88-b3e7-3599fe9cbfd9" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b" exitCode=255
Nov 02 14:55:47.949546 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.949530    5041 manager.go:324] Recovery completed
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.972870    5041 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.972897    5041 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.972926    5041 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973099    5041 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973113    5041 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973137    5041 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973145    5041 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973153    5041 policy_none.go:50] "Start"
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973171    5041 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:55:47.973386 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973183    5041 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:55:47.973851 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973413    5041 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 14:55:47.973851 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973425    5041 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 14:55:47.973851 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.973433    5041 policy_none.go:44] "Start"
Nov 02 14:55:47.978853 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.978828    5041 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:55:47.979356 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:47.979342    5041 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 14:55:47.982986 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.982974    5041 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:55:47.983645 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:47.982999    5041 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:55:47.983708 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.983648    5041 server.go:87] "Starting device plugin registration server"
Nov 02 14:55:47.983974 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.983963    5041 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:55:47.984027 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.983978    5041 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:55:47.984112 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.984098    5041 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:55:47.984211 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.984185    5041 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:55:47.984262 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.984219    5041 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:55:47.984262 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.984227    5041 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:55:47.984352 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.984272    5041 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:55:47.984209427Z","UUID":"69e14319-5196-4cf1-87c4-f2b4b2a58031","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:55:47.985602 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:47.985580    5041 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:55:47.985886 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.985867    5041 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:55:47.986101 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:47.985910    5041 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:55:48.084477 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.084454    5041 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:48.085706 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.085690    5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:48.085777 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.085717    5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:48.085777 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.085732    5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-143.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:48.085861 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.085781    5041 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.120074 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.120055    5041 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.120139 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.120111    5041 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.180308 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180262    5041 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"]
Nov 02 14:55:48.180870 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180839    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" event={"ID":"041ae145e73ad7d91d16b7f00a767ff7","Type":"ContainerStarted","Data":"bfb7ec3d524e7f1c20d6502e0dc193f66506d86cf423a034e51f75540eb1719a"}
Nov 02 14:55:48.180949 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180878    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" event={"ID":"041ae145e73ad7d91d16b7f00a767ff7","Type":"ContainerDied","Data":"20c7b4e8d1e5af794c7935612b033a7e8ed191eb5b905cb4202f8a256df554f3"}
Nov 02 14:55:48.180949 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180900    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal" event={"ID":"041ae145e73ad7d91d16b7f00a767ff7","Type":"ContainerStarted","Data":"124363fbae2f80f8f1f1a45a9d9d845db481127ee3310eb93205a59798cb6894"}
Nov 02 14:55:48.180949 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180944    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.181090 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" event={"ID":"15b2f7b139a98a51bb6e53a9a5605774","Type":"ContainerStarted","Data":"de3fa8f50d29ae7c46dc9b39362b1bf3553958f80003dce6944ed786a73a99d1"}
Nov 02 14:55:48.181090 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.180968    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" event={"ID":"15b2f7b139a98a51bb6e53a9a5605774","Type":"ContainerStarted","Data":"7b3e6d9c88e8b73fd587e90e5ef313b50b03b9ef5410056a823df4f0af92bf66"}
Nov 02 14:55:48.181090 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.181020    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.208529 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.208509    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:48.208598 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:48.208545    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.209052 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.209036    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:48.209130 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:48.209077    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.294174 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.294151    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.294241 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.294179    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.294241 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.294196    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15b2f7b139a98a51bb6e53a9a5605774-config\") pod \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" (UID: \"15b2f7b139a98a51bb6e53a9a5605774\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.394751 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.394720    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.394751 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.394753    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15b2f7b139a98a51bb6e53a9a5605774-config\") pod \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" (UID: \"15b2f7b139a98a51bb6e53a9a5605774\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.394907 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.394768    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.394907 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.394822    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.394907 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.394844    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15b2f7b139a98a51bb6e53a9a5605774-config\") pod \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" (UID: \"15b2f7b139a98a51bb6e53a9a5605774\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.394907 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.394822    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041ae145e73ad7d91d16b7f00a767ff7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" (UID: \"041ae145e73ad7d91d16b7f00a767ff7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.826910 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.826881    5041 apiserver.go:52] "Watching apiserver"
Nov 02 14:55:48.877715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.877684    5041 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:55:48.878445 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.878417    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhlcp","openshift-cluster-node-tuning-operator/tuned-j2xpj","openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29","openshift-multus/multus-additional-cni-plugins-72q2z","openshift-network-console/networking-console-plugin-86c478bb88-cfnwj","openshift-ovn-kubernetes/ovnkube-node-wjtlf","openshift-insights/insights-runtime-extractor-rzg46","openshift-insights/periodic-gathering-j6hws-tdl49","openshift-network-diagnostics/network-check-target-gnfc9","openshift-image-registry/image-registry-84f8679cb6-t6942","kube-system/global-pull-secret-syncer-gc6sk","openshift-ingress/router-default-659675459f-cp5t4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal","openshift-network-operator/iptables-alerter-jknvp","openshift-service-ca/service-ca-5fffc97bdf-b89mz","kube-system/konnectivity-agent-svzrl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz","openshift-ingress-canary/ingress-canary-d8jgh","openshift-multus/multus-6f4vs","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th","openshift-insights/insights-operator-7c4986d449-8nlz6","kube-system/kas-connection-checker-685895c7c9-zp66b","kube-system/kas-connection-checker-685895c7c9-bt2l4","kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42","openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx","openshift-network-diagnostics/network-check-source-6f78d59f99-hv272","openshift-console-operator/console-operator-7cd56b8dbc-jlsqw","openshift-dns/node-resolver-g4d2h","openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs","openshift-multus/network-metrics-daemon-jfqf9","openshift-dns/dns-default-swqf5","openshift-image-registry/image-registry-588df7d7d7-sv5v5","openshift-image-registry/node-ca-vm8rc","openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"]
Nov 02 14:55:48.879818 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.879796    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:48.880657 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.880636    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:48.881339 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.881323    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:48.882089 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.882071    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.882794 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.882776    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:48.886399 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.886373    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:48.888031 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.888014    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:48.891755 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.891738    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.893874 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.893857    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.894793 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.894774    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 14:55:48.895001 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.894986    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 14:55:48.895048 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.895039    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.895113 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.895103    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.895206 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.895196    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.895270 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.895254    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.896313 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.896293    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 14:55:48.902399 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902382    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.902461 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902419    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 14:55:48.902461 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902449    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.902655 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902641    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 14:55:48.902698 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902647    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 14:55:48.902698 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902665    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.902942 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902919    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.903028 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902965    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 14:55:48.903028 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902993    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 14:55:48.903028 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.903004    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 14:55:48.903028 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.902999    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.906321 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.906309    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 14:55:48.907429 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907388    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 14:55:48.907429 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907425    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 14:55:48.907596 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907442    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 14:55:48.907657 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907605    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 14:55:48.907657 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907605    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907687    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907693    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907686    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907709    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907731    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907758    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907766    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907810    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 14:55:48.907812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907817    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.908183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907782    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.908183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907880    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 14:55:48.908183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907945    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.908183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.907970    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.908183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.908100    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-phchw\""
Nov 02 14:55:48.908183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.908149    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 14:55:48.909436 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.909419    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:55:48.909532 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.909488    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:55:48.909614 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.909594    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:55:48.909983 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.909961    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 14:55:48.910563 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910544    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.910645 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910627    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:55:48.910685 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910642    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.910749 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910610    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.910864 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910820    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8vd5b\""
Nov 02 14:55:48.910937 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910882    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 14:55:48.910988 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910572    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dmmqr\""
Nov 02 14:55:48.910988 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.910981    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.911138 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911093    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 14:55:48.911219 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911198    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-25j7m\""
Nov 02 14:55:48.911497 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911479    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pvkf\""
Nov 02 14:55:48.911635 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911552    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.911635 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911604    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:55:48.911739 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911691    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:55:48.911964 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911933    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:55:48.912084 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911993    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kws2w\""
Nov 02 14:55:48.912084 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912001    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:55:48.912084 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912057    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:55:48.912084 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912077    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 14:55:48.912348 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.911941    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x78pf\""
Nov 02 14:55:48.912348 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912207    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczs\""
Nov 02 14:55:48.912348 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912231    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.912348 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912269    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:55:48.912958 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912939    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-st89p\""
Nov 02 14:55:48.913055 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.912981    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.913055 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913011    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.913162 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913122    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:55:48.913211 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913124    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.913257 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913211    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.913439 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913425    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:55:48.913521 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913440    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.913521 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913492    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:55:48.913664 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913560    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.913664 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913625    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.913753 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913563    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.913753 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913694    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.913753 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913708    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.913753 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913726    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:55:48.914007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913825    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62zsq\""
Nov 02 14:55:48.914007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913856    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:55:48.914007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913938    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:55:48.914007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913710    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:55:48.914007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913990    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.914007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.913998    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:55:48.914235 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914028    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jmg9z\""
Nov 02 14:55:48.914235 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914033    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.914235 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914041    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:55:48.914235 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914066    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.914235 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914094    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:55:48.914235 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914118    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tsm8f\""
Nov 02 14:55:48.914563 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914296    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-czhhz\""
Nov 02 14:55:48.914563 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914301    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7sjw\""
Nov 02 14:55:48.914563 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914317    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 14:55:48.914744 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914727    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-btkxl\""
Nov 02 14:55:48.914744 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914736    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2fmch\""
Nov 02 14:55:48.914958 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914851    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.914958 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.914934    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:55:48.915041 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.915008    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pj6nr\""
Nov 02 14:55:48.918271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.918252    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 14:55:48.918271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.918266    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 14:55:48.918408 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.918304    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 14:55:48.918408 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.918330    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 14:55:48.918408 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.918340    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 14:55:48.918408 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.918350    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rrjlf\""
Nov 02 14:55:48.942153 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.942136    5041 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:55:48.944100 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.944085    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.944231 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.944218    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.966286 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.966269    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:48.966430 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:48.966310    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-143.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.967305 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.967293    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:48.967392 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:48.967324    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-143.ec2.internal"
Nov 02 14:55:48.996871 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996851    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-script-lib\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.996942 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996875    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-snapshots\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:48.996942 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996892    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwq98\" (UniqueName: \"kubernetes.io/projected/127445ba-9a8b-4291-931e-4f2f5a24a41d-kube-api-access-nwq98\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:48.996942 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996910    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-conf\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.996942 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996932    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-tmp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.997121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996956    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:48.997121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.996995    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-snapshots\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:48.997121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997015    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxwm\" (UniqueName: \"kubernetes.io/projected/dc383de8-7d40-46ee-84ff-74b20040ff2c-kube-api-access-9pxwm\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:48.997121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997053    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-tmp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.997121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:48.997121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997108    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-daemon-config\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.997410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997141    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-etc-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.997410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997164    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.997410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997188    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:48.997410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997202    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.997410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997339    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.997410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997395    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-socket-dir-parent\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/485e5e3f-08d1-44e0-b93b-1023a530c305-iptables-alerter-script\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997528    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vhh\" (UniqueName: \"kubernetes.io/projected/e6beba97-264e-4107-bafa-57b5380ad218-kube-api-access-m4vhh\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997548    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-daemon-config\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997570    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-script-lib\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997576    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-tmp-dir\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997621    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-tmp-dir\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997627    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997655    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-dbus\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:48.997686 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997683    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997707    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5150492-44d2-426a-a833-69856c7ddf09-crio-socket\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997735    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6831662e-48ae-42bc-95b4-27633e745504-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997751    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/485e5e3f-08d1-44e0-b93b-1023a530c305-iptables-alerter-script\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997764    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997864    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997890    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-sys\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997911    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6831662e-48ae-42bc-95b4-27633e745504-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997920    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-var-lib-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997943    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-log-socket\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997977    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4grh\" (UniqueName: \"kubernetes.io/projected/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-kube-api-access-m4grh\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.997999    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-systemd\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998024    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzpw\" (UniqueName: \"kubernetes.io/projected/f523968a-ce21-493a-80f9-f02a142a320a-kube-api-access-7tzpw\") pod \"network-check-source-6f78d59f99-hv272\" (UID: \"f523968a-ce21-493a-80f9-f02a142a320a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998052    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998055    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:48.998127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998104    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-device-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998183    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzrp\" (UniqueName: \"kubernetes.io/projected/717989db-0433-4e65-95a3-6a367b4e682b-kube-api-access-hvzrp\") pod \"migrator-66df5d496b-kn6nx\" (UID: \"717989db-0433-4e65-95a3-6a367b4e682b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998218    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-cabundle\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998241    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998257    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx979\" (UniqueName: \"kubernetes.io/projected/f57d8416-b76f-4dbf-8fd8-3e43d001e063-kube-api-access-kx979\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998275    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba225e36-3f26-42cf-b829-4e3117f7e899-host\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998302    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998396    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-config-volume\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998422    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998448    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqkl\" (UniqueName: \"kubernetes.io/projected/e5150492-44d2-426a-a833-69856c7ddf09-kube-api-access-gvqkl\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998448    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-cabundle\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998472    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-node-log\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998472    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998495    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-config\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998533    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc383de8-7d40-46ee-84ff-74b20040ff2c-serving-cert\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998599    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998601    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-config-volume\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-kubernetes\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998680    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-ovnkube-config\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998700    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxw2p\" (UniqueName: \"kubernetes.io/projected/9f189efb-95a0-4e40-9b7b-8076e77c5508-kube-api-access-bxw2p\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.998707 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998717    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dr6\" (UniqueName: \"kubernetes.io/projected/113ca502-9379-4a88-b3e7-3599fe9cbfd9-kube-api-access-p8dr6\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998733    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/485e5e3f-08d1-44e0-b93b-1023a530c305-host-slash\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998753    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5150492-44d2-426a-a833-69856c7ddf09-data-volume\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998844    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc383de8-7d40-46ee-84ff-74b20040ff2c-serving-cert\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998871    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5150492-44d2-426a-a833-69856c7ddf09-data-volume\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998919    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-tuned\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998937    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-cnibin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998967    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-cni-binary-copy\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.998990    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-system-cni-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999023    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-tuned\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999032    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-binary-copy\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999075    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999141    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f189efb-95a0-4e40-9b7b-8076e77c5508-cni-binary-copy\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999176    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c08878-17e0-4207-8b21-a3e3da129d37-serving-cert\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-binary-copy\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999327    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999351    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999384    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999411    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999443    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph8h\" (UniqueName: \"kubernetes.io/projected/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-kube-api-access-fph8h\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:48.999478 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999471    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999553    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-host\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999585    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8fkp\" (UniqueName: \"kubernetes.io/projected/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-kube-api-access-t8fkp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999607    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c08878-17e0-4207-8b21-a3e3da129d37-serving-cert\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999615    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-multus-certs\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999609    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999646    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-bin\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999668    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999685    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-config\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999794    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6432dfb-7757-4a30-a440-8cc647505f76-konnectivity-ca\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999833    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lsp\" (UniqueName: \"kubernetes.io/projected/c9af00a4-820f-473f-bc2d-7141fbe02975-kube-api-access-z2lsp\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999860    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb5081a6-d3c8-4971-9e46-49f330c7913f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:48.999887    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000033    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113ca502-9379-4a88-b3e7-3599fe9cbfd9-config\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000049    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf141b7d-a42e-46d1-b71b-987f2c080604-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000059    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453f0247-70c9-4b6c-9272-afe056657e5d-original-pull-secret\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000068    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-lib-modules\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000078    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6432dfb-7757-4a30-a440-8cc647505f76-konnectivity-ca\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000085    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000123    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb5081a6-d3c8-4971-9e46-49f330c7913f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000179    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000268    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.000271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000270    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf141b7d-a42e-46d1-b71b-987f2c080604-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000296    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000416    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-sys-fs\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000430    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000449    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysconfig\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000494    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-var-lib-kubelet\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000520    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-k8s-cni-cncf-io\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000547    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d71694a7-787c-45ec-9308-6ff029eead3f-hosts-file\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000545    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000588    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000620    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78vt\" (UniqueName: \"kubernetes.io/projected/86f99765-b728-464e-bce3-fb178f34d0ed-kube-api-access-j78vt\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000712    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphv8\" (UniqueName: \"kubernetes.io/projected/485e5e3f-08d1-44e0-b93b-1023a530c305-kube-api-access-rphv8\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000764    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000799    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc383de8-7d40-46ee-84ff-74b20040ff2c-config\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:49.000877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000869    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-socket-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.000992    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdnc\" (UniqueName: \"kubernetes.io/projected/4678cafc-05c2-429b-83b6-3caa694c7d0f-kube-api-access-bwdnc\") pod \"volume-data-source-validator-58c9f5f46d-w84hz\" (UID: \"4678cafc-05c2-429b-83b6-3caa694c7d0f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-default-certificate\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001011    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc383de8-7d40-46ee-84ff-74b20040ff2c-config\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001028    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-key\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001044    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-bin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001058    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-kubelet\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001080    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-kubelet-config\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001113    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrwz\" (UniqueName: \"kubernetes.io/projected/f54d3276-9a6f-498e-84ff-a48b1651bc09-kube-api-access-bhrwz\") pod \"kas-connection-checker-685895c7c9-zp66b\" (UID: \"f54d3276-9a6f-498e-84ff-a48b1651bc09\") " pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001133    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001255    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/127445ba-9a8b-4291-931e-4f2f5a24a41d-signing-key\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001257    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-multus\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001305    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfr8\" (UniqueName: \"kubernetes.io/projected/ba12693c-0a77-49d5-a792-d420be330a7f-kube-api-access-cpfr8\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001356    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001391    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-default-certificate\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.001400 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001402    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrlp\" (UniqueName: \"kubernetes.io/projected/d71694a7-787c-45ec-9308-6ff029eead3f-kube-api-access-7xrlp\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001435    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fqg\" (UniqueName: \"kubernetes.io/projected/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kube-api-access-b6fqg\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001462    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-netns\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001483    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001488    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af00a4-820f-473f-bc2d-7141fbe02975-cert\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001531    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f57d8416-b76f-4dbf-8fd8-3e43d001e063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001533    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n2x\" (UniqueName: \"kubernetes.io/projected/eb5081a6-d3c8-4971-9e46-49f330c7913f-kube-api-access-m4n2x\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001601    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89m4\" (UniqueName: \"kubernetes.io/projected/60554d74-3158-455b-bfef-2d8b4e9e6feb-kube-api-access-h89m4\") pod \"kas-connection-checker-685895c7c9-bt2l4\" (UID: \"60554d74-3158-455b-bfef-2d8b4e9e6feb\") " pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001664    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9af00a4-820f-473f-bc2d-7141fbe02975-cert\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001655    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-tmp\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001717    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-system-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001775    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-kubelet\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001749    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c08878-17e0-4207-8b21-a3e3da129d37-tmp\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001800    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-env-overrides\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001839    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba12693c-0a77-49d5-a792-d420be330a7f-ovn-node-metrics-cert\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001856    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d71694a7-787c-45ec-9308-6ff029eead3f-tmp-dir\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001908    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-registration-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001925    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-etc-selinux\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d71694a7-787c-45ec-9308-6ff029eead3f-tmp-dir\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001943    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001968    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.001967    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-service-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba12693c-0a77-49d5-a792-d420be330a7f-env-overrides\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002016    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcfn\" (UniqueName: \"kubernetes.io/projected/bca16489-34d0-4230-b21a-7ea110ff8c7f-kube-api-access-kpcfn\") pod \"kas-connection-checker-685895c7c9-dhlcp\" (UID: \"bca16489-34d0-4230-b21a-7ea110ff8c7f\") " pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002027    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba12693c-0a77-49d5-a792-d420be330a7f-ovn-node-metrics-cert\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002035    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcf9\" (UniqueName: \"kubernetes.io/projected/ba225e36-3f26-42cf-b829-4e3117f7e899-kube-api-access-fhcf9\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002058    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cnibin\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002088    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.002087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002102    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c08878-17e0-4207-8b21-a3e3da129d37-service-ca-bundle\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87z87\" (UniqueName: \"kubernetes.io/projected/40c08878-17e0-4207-8b21-a3e3da129d37-kube-api-access-87z87\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002180    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-modprobe-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002198    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-systemd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002226    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002255    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002287    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-run\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002334    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002395    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-conf-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002423    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-systemd-units\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002446    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002516    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wgv\" (UniqueName: \"kubernetes.io/projected/05e6fe27-aaeb-479c-9075-7590e278d505-kube-api-access-x5wgv\") pod \"migrator-66df5d496b-kg6gs\" (UID: \"05e6fe27-aaeb-479c-9075-7590e278d505\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002588    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-ovn\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002605    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-netd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002624    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002641    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002667    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6831662e-48ae-42bc-95b4-27633e745504-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002692    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-netns\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002747    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002837    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002840    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-stats-auth\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002946    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-hostroot\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002969    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.002998    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-etc-kubernetes\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003032    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003051    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113ca502-9379-4a88-b3e7-3599fe9cbfd9-serving-cert\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003067    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba225e36-3f26-42cf-b829-4e3117f7e899-serviceca\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003084    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-stats-auth\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003089    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6432dfb-7757-4a30-a440-8cc647505f76-agent-certs\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003122    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003151    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk8k\" (UniqueName: \"kubernetes.io/projected/6831662e-48ae-42bc-95b4-27633e745504-kube-api-access-jrk8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003119    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6831662e-48ae-42bc-95b4-27633e745504-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003165    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003196    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-slash\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003205    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113ca502-9379-4a88-b3e7-3599fe9cbfd9-serving-cert\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003223    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003264    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba225e36-3f26-42cf-b829-4e3117f7e899-serviceca\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003385    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6432dfb-7757-4a30-a440-8cc647505f76-agent-certs\") pod \"konnectivity-agent-svzrl\" (UID: \"a6432dfb-7757-4a30-a440-8cc647505f76\") " pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:55:49.003410 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.003409    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:49.025307 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.025241    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.025554 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.025531    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vhh\" (UniqueName: \"kubernetes.io/projected/e6beba97-264e-4107-bafa-57b5380ad218-kube-api-access-m4vhh\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.025654 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.025564    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4grh\" (UniqueName: \"kubernetes.io/projected/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-kube-api-access-m4grh\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:49.025654 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.025641    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxw2p\" (UniqueName: \"kubernetes.io/projected/9f189efb-95a0-4e40-9b7b-8076e77c5508-kube-api-access-bxw2p\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.026027 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.025644    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqkl\" (UniqueName: \"kubernetes.io/projected/e5150492-44d2-426a-a833-69856c7ddf09-kube-api-access-gvqkl\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.026027 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.025930    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.026027 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026007    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwq98\" (UniqueName: \"kubernetes.io/projected/127445ba-9a8b-4291-931e-4f2f5a24a41d-kube-api-access-nwq98\") pod \"service-ca-5fffc97bdf-b89mz\" (UID: \"127445ba-9a8b-4291-931e-4f2f5a24a41d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b89mz"
Nov 02 14:55:49.026027 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026018    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dr6\" (UniqueName: \"kubernetes.io/projected/113ca502-9379-4a88-b3e7-3599fe9cbfd9-kube-api-access-p8dr6\") pod \"console-operator-7cd56b8dbc-jlsqw\" (UID: \"113ca502-9379-4a88-b3e7-3599fe9cbfd9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:49.026271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026177    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxwm\" (UniqueName: \"kubernetes.io/projected/dc383de8-7d40-46ee-84ff-74b20040ff2c-kube-api-access-9pxwm\") pod \"service-ca-operator-544d749c8c-42nrz\" (UID: \"dc383de8-7d40-46ee-84ff-74b20040ff2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz"
Nov 02 14:55:49.026271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026186    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzrp\" (UniqueName: \"kubernetes.io/projected/717989db-0433-4e65-95a3-6a367b4e682b-kube-api-access-hvzrp\") pod \"migrator-66df5d496b-kn6nx\" (UID: \"717989db-0433-4e65-95a3-6a367b4e682b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kn6nx"
Nov 02 14:55:49.026271 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026225    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8fkp\" (UniqueName: \"kubernetes.io/projected/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-kube-api-access-t8fkp\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.026462 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026388    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lsp\" (UniqueName: \"kubernetes.io/projected/c9af00a4-820f-473f-bc2d-7141fbe02975-kube-api-access-z2lsp\") pod \"ingress-canary-d8jgh\" (UID: \"c9af00a4-820f-473f-bc2d-7141fbe02975\") " pod="openshift-ingress-canary/ingress-canary-d8jgh"
Nov 02 14:55:49.026559 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026536    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzpw\" (UniqueName: \"kubernetes.io/projected/f523968a-ce21-493a-80f9-f02a142a320a-kube-api-access-7tzpw\") pod \"network-check-source-6f78d59f99-hv272\" (UID: \"f523968a-ce21-493a-80f9-f02a142a320a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hv272"
Nov 02 14:55:49.026559 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx979\" (UniqueName: \"kubernetes.io/projected/f57d8416-b76f-4dbf-8fd8-3e43d001e063-kube-api-access-kx979\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.026690 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026604    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph8h\" (UniqueName: \"kubernetes.io/projected/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-kube-api-access-fph8h\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:49.026769 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026748    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") pod \"periodic-gathering-j6hws-tdl49\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") " pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:55:49.026877 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.026863    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.030862 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.030846    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.032939 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.032924    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphv8\" (UniqueName: \"kubernetes.io/projected/485e5e3f-08d1-44e0-b93b-1023a530c305-kube-api-access-rphv8\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:49.033414 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.033399    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78vt\" (UniqueName: \"kubernetes.io/projected/86f99765-b728-464e-bce3-fb178f34d0ed-kube-api-access-j78vt\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.035993 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.035956    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrwz\" (UniqueName: \"kubernetes.io/projected/f54d3276-9a6f-498e-84ff-a48b1651bc09-kube-api-access-bhrwz\") pod \"kas-connection-checker-685895c7c9-zp66b\" (UID: \"f54d3276-9a6f-498e-84ff-a48b1651bc09\") " pod="kube-system/kas-connection-checker-685895c7c9-zp66b"
Nov 02 14:55:49.036626 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95knt\" (UniqueName: \"kubernetes.io/projected/4709665b-4994-4036-836f-225e49ad8233-kube-api-access-95knt\") pod \"network-check-target-gnfc9\" (UID: \"4709665b-4994-4036-836f-225e49ad8233\") " pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:49.036626 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036183    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdnc\" (UniqueName: \"kubernetes.io/projected/4678cafc-05c2-429b-83b6-3caa694c7d0f-kube-api-access-bwdnc\") pod \"volume-data-source-validator-58c9f5f46d-w84hz\" (UID: \"4678cafc-05c2-429b-83b6-3caa694c7d0f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w84hz"
Nov 02 14:55:49.036626 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036399    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wgv\" (UniqueName: \"kubernetes.io/projected/05e6fe27-aaeb-479c-9075-7590e278d505-kube-api-access-x5wgv\") pod \"migrator-66df5d496b-kg6gs\" (UID: \"05e6fe27-aaeb-479c-9075-7590e278d505\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kg6gs"
Nov 02 14:55:49.036626 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036405    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89m4\" (UniqueName: \"kubernetes.io/projected/60554d74-3158-455b-bfef-2d8b4e9e6feb-kube-api-access-h89m4\") pod \"kas-connection-checker-685895c7c9-bt2l4\" (UID: \"60554d74-3158-455b-bfef-2d8b4e9e6feb\") " pod="kube-system/kas-connection-checker-685895c7c9-bt2l4"
Nov 02 14:55:49.036626 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036588    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfr8\" (UniqueName: \"kubernetes.io/projected/ba12693c-0a77-49d5-a792-d420be330a7f-kube-api-access-cpfr8\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.036862 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036633    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87z87\" (UniqueName: \"kubernetes.io/projected/40c08878-17e0-4207-8b21-a3e3da129d37-kube-api-access-87z87\") pod \"insights-operator-7c4986d449-8nlz6\" (UID: \"40c08878-17e0-4207-8b21-a3e3da129d37\") " pod="openshift-insights/insights-operator-7c4986d449-8nlz6"
Nov 02 14:55:49.036862 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.036704    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fqg\" (UniqueName: \"kubernetes.io/projected/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kube-api-access-b6fqg\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.038080 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.038061    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcf9\" (UniqueName: \"kubernetes.io/projected/ba225e36-3f26-42cf-b829-4e3117f7e899-kube-api-access-fhcf9\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:49.038662 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.038646    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrlp\" (UniqueName: \"kubernetes.io/projected/d71694a7-787c-45ec-9308-6ff029eead3f-kube-api-access-7xrlp\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.041095 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.041072    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n2x\" (UniqueName: \"kubernetes.io/projected/eb5081a6-d3c8-4971-9e46-49f330c7913f-kube-api-access-m4n2x\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.041179 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.041164    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcfn\" (UniqueName: \"kubernetes.io/projected/bca16489-34d0-4230-b21a-7ea110ff8c7f-kube-api-access-kpcfn\") pod \"kas-connection-checker-685895c7c9-dhlcp\" (UID: \"bca16489-34d0-4230-b21a-7ea110ff8c7f\") " pod="kube-system/kas-connection-checker-685895c7c9-dhlcp"
Nov 02 14:55:49.041179 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.041175    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk8k\" (UniqueName: \"kubernetes.io/projected/6831662e-48ae-42bc-95b4-27633e745504-kube-api-access-jrk8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v5k42\" (UID: \"6831662e-48ae-42bc-95b4-27633e745504\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42"
Nov 02 14:55:49.104202 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104139    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d71694a7-787c-45ec-9308-6ff029eead3f-hosts-file\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.104202 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104175    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-socket-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.104202 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104199    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-bin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104219    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-kubelet\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104240    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-kubelet-config\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104265    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-multus\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104264    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-socket-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-bin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104290    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-netns\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104290    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-kubelet\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104309    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-kubelet-config\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104316    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-system-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d71694a7-787c-45ec-9308-6ff029eead3f-hosts-file\") pod \"node-resolver-g4d2h\" (UID: \"d71694a7-787c-45ec-9308-6ff029eead3f\") " pod="openshift-dns/node-resolver-g4d2h"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-var-lib-cni-multus\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104332    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-netns\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104351    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-kubelet\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.104423 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104398    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-system-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-kubelet\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-registration-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104499    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-registration-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104523    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-etc-selinux\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104563    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104600    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cnibin\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104572    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-etc-selinux\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104623    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104651    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-modprobe-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104657    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104667    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-cnibin\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.104679    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.604659499 +0000 UTC m=+1.862728954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104706    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-systemd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104728    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104747    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-modprobe-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104752    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-run\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104760    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-systemd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104782    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-conf-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104793    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-run\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104807    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-systemd-units\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.104825 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104834    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-ovn\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104841    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-conf-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104786    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104858    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-netd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104889    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-systemd-units\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104885    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-run-ovn\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104895    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-netns\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104916    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-netns\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104925    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-netd\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104939    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-hostroot\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104964    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-etc-kubernetes\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105016    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-etc-kubernetes\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105018    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-slash\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.104998    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-hostroot\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105043    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105069    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-conf\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105082    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-slash\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105096    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-etc-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105126    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105140    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105161    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-etc-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105165    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105189    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-socket-dir-parent\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105195    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105216    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105219    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-conf\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105246    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-dbus\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105273    5041 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105301    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453f0247-70c9-4b6c-9272-afe056657e5d-dbus\") pod \"global-pull-secret-syncer-gc6sk\" (UID: \"453f0247-70c9-4b6c-9272-afe056657e5d\") " pod="kube-system/global-pull-secret-syncer-gc6sk"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105307    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysctl-d\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105301    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-socket-dir-parent\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105321    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5150492-44d2-426a-a833-69856c7ddf09-crio-socket\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105350    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105388    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105392    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5150492-44d2-426a-a833-69856c7ddf09-crio-socket\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105411    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-sys\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105446    5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105456    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-sys\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105461    5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588df7d7d7-sv5v5: secret "image-registry-tls" not found
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105467    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-var-lib-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-var-lib-openvswitch\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105514    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-log-socket\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105451    5041 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105544    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls podName:58e33558-be4f-4b8b-812e-f73322516ac0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.605529165 +0000 UTC m=+1.863598608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls") pod "image-registry-588df7d7d7-sv5v5" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0") : secret "image-registry-tls" not found
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105546    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-log-socket\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105567    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-systemd\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105591    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.605576699 +0000 UTC m=+1.863646128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105608    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-systemd\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105619    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-device-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105645    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba225e36-3f26-42cf-b829-4e3117f7e899-host\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105668    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105692    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-node-log\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105695    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-device-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105719    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105734    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba225e36-3f26-42cf-b829-4e3117f7e899-host\") pod \"node-ca-vm8rc\" (UID: \"ba225e36-3f26-42cf-b829-4e3117f7e899\") " pod="openshift-image-registry/node-ca-vm8rc"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105744    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105765    5041 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105768    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-kubernetes\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105782    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-node-log\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105794    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/485e5e3f-08d1-44e0-b93b-1023a530c305-host-slash\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:49.105763 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105812    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.60580066 +0000 UTC m=+1.863870132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105856    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/485e5e3f-08d1-44e0-b93b-1023a530c305-host-slash\") pod \"iptables-alerter-jknvp\" (UID: \"485e5e3f-08d1-44e0-b93b-1023a530c305\") " pod="openshift-network-operator/iptables-alerter-jknvp"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105864    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.105902    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.605889404 +0000 UTC m=+1.863958833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105899    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-kubernetes\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105917    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105939    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-cnibin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105962    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-system-cni-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.105987    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-multus-cni-dir\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106007    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106020    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-cnibin\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106032    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106057    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-host\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106065    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f57d8416-b76f-4dbf-8fd8-3e43d001e063-system-cni-dir\") pod \"multus-additional-cni-plugins-72q2z\" (UID: \"f57d8416-b76f-4dbf-8fd8-3e43d001e063\") " pod="openshift-multus/multus-additional-cni-plugins-72q2z"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106079    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-multus-certs\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.106110    5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106118    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-multus-certs\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.106123    5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f8679cb6-t6942: secret "image-registry-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.106159    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls podName:0f276b8e-7e4f-4980-a8a3-9fed6ee92685 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.60615004 +0000 UTC m=+1.864219506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls") pod "image-registry-84f8679cb6-t6942" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685") : secret "image-registry-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106179    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-host\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106200    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106214    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-bin\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106246    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba12693c-0a77-49d5-a792-d420be330a7f-host-cni-bin\") pod \"ovnkube-node-wjtlf\" (UID: \"ba12693c-0a77-49d5-a792-d420be330a7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106263    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-lib-modules\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106291    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106334    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-sys-fs\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.106414    5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.106450    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:49.606439427 +0000 UTC m=+1.864508858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106449    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870c8c3-b2d0-4cd6-b2f7-7eec96149b97-sys-fs\") pod \"aws-ebs-csi-driver-node-njb9f\" (UID: \"4870c8c3-b2d0-4cd6-b2f7-7eec96149b97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njb9f"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106357    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysconfig\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106497    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-lib-modules\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106507    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-var-lib-kubelet\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106536    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-etc-sysconfig\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-k8s-cni-cncf-io\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106630    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee-var-lib-kubelet\") pod \"tuned-j2xpj\" (UID: \"ad2c0eec-9b2e-4b78-b96d-36b37dd1e2ee\") " pod="openshift-cluster-node-tuning-operator/tuned-j2xpj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.106661    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f189efb-95a0-4e40-9b7b-8076e77c5508-host-run-k8s-cni-cncf-io\") pod \"multus-6f4vs\" (UID: \"9f189efb-95a0-4e40-9b7b-8076e77c5508\") " pod="openshift-multus/multus-6f4vs"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.107933    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3be13e2-c47a-4a45-b023-fff5e1ec32ee-metrics-certs\") pod \"network-metrics-daemon-jfqf9\" (UID: \"a3be13e2-c47a-4a45-b023-fff5e1ec32ee\") " pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.108381    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/86f99765-b728-464e-bce3-fb178f34d0ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l66th\" (UID: \"86f99765-b728-464e-bce3-fb178f34d0ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.108821    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf141b7d-a42e-46d1-b71b-987f2c080604-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cfnwj\" (UID: \"bf141b7d-a42e-46d1-b71b-987f2c080604\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:49.109325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.109115    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ca09ed6-20a8-4b83-bae6-a371cbf71d65-metrics-tls\") pod \"dns-default-swqf5\" (UID: \"8ca09ed6-20a8-4b83-bae6-a371cbf71d65\") " pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:49.180067 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.180049    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"
Nov 02 14:55:49.181071 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.181054    5041 scope.go:122] "RemoveContainer" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b"
Nov 02 14:55:49.183045 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.183030    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"
Nov 02 14:55:49.186772 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.186756    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfqf9"
Nov 02 14:55:49.191985 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.191968    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:49.494440 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.494411    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cfnwj"]
Nov 02 14:55:49.498398 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:49.498352    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf141b7d_a42e_46d1_b71b_987f2c080604.slice/crio-4faf9177425b6f332fdf12912daf5edf8a879b1faf26934c09a7ecf67bea812a WatchSource:0}: Error finding container 4faf9177425b6f332fdf12912daf5edf8a879b1faf26934c09a7ecf67bea812a: Status 404 returned error can't find the container with id 4faf9177425b6f332fdf12912daf5edf8a879b1faf26934c09a7ecf67bea812a
Nov 02 14:55:49.500515 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.500497    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:55:49.506626 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.506608    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th"]
Nov 02 14:55:49.528405 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.528384    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jfqf9"]
Nov 02 14:55:49.531274 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:49.531248    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3be13e2_c47a_4a45_b023_fff5e1ec32ee.slice/crio-a03d1f9a1f3b1521f4e9d32a61ecdcf00fa384ba83a7865b473bfc766190959c WatchSource:0}: Error finding container a03d1f9a1f3b1521f4e9d32a61ecdcf00fa384ba83a7865b473bfc766190959c: Status 404 returned error can't find the container with id a03d1f9a1f3b1521f4e9d32a61ecdcf00fa384ba83a7865b473bfc766190959c
Nov 02 14:55:49.532272 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.532249    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-swqf5"]
Nov 02 14:55:49.545249 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:49.545226    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca09ed6_20a8_4b83_bae6_a371cbf71d65.slice/crio-0470e194ffd5f56ac2a59943815e04c4df0089015b2c45c1ce827e6a0fae3493 WatchSource:0}: Error finding container 0470e194ffd5f56ac2a59943815e04c4df0089015b2c45c1ce827e6a0fae3493: Status 404 returned error can't find the container with id 0470e194ffd5f56ac2a59943815e04c4df0089015b2c45c1ce827e6a0fae3493
Nov 02 14:55:49.610786 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.610764    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.610875 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.610791    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.610875 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.610816    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.610875 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.610860    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.610884    5041 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.610899    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.610929    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:50.610913871 +0000 UTC m=+2.868983320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.610956    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.610969    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.611008    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:50.610998248 +0000 UTC m=+2.869067676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:49.611035 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.611022    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:49.611383 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.611041    5041 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:49.611383 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.611051    5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:49.611383 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.611083    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:50.611072029 +0000 UTC m=+2.869141467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:49.611383 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.611097    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:50.61109097 +0000 UTC m=+2.869160401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:49.611383 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.611151    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:50.611141006 +0000 UTC m=+2.869210453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:49.613064 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.613047    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"image-registry-588df7d7d7-sv5v5\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") " pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.613121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.613080    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"image-registry-84f8679cb6-t6942\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") " pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.665809 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.665788    5041 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:49.782595 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.782534    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:49.788482 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.788459    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:49.913696 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.913642    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:49.949108 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.949077    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj" event={"ID":"bf141b7d-a42e-46d1-b71b-987f2c080604","Type":"ContainerStarted","Data":"4faf9177425b6f332fdf12912daf5edf8a879b1faf26934c09a7ecf67bea812a"}
Nov 02 14:55:49.950615 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.950595    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 14:55:49.950992 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.950974    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/1.log"
Nov 02 14:55:49.951084 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.951003    5041 generic.go:356] "Generic (PLEG): container finished" podID="113ca502-9379-4a88-b3e7-3599fe9cbfd9" containerID="92071144c8b133f87b71ef086c56c6899ac0b5167d2edc9d86941a32dbc1f610" exitCode=255
Nov 02 14:55:49.951084 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.951071    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" event={"ID":"113ca502-9379-4a88-b3e7-3599fe9cbfd9","Type":"ContainerDied","Data":"92071144c8b133f87b71ef086c56c6899ac0b5167d2edc9d86941a32dbc1f610"}
Nov 02 14:55:49.951170 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.951115    5041 scope.go:122] "RemoveContainer" containerID="2b2556a46aef2d6f6e322476879be52d3894e9783766b5a86e4c1118c305cd8b"
Nov 02 14:55:49.951318 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.951299    5041 scope.go:122] "RemoveContainer" containerID="92071144c8b133f87b71ef086c56c6899ac0b5167d2edc9d86941a32dbc1f610"
Nov 02 14:55:49.951537 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:49.951518    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlsqw_openshift-console-operator(113ca502-9379-4a88-b3e7-3599fe9cbfd9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" podUID="113ca502-9379-4a88-b3e7-3599fe9cbfd9"
Nov 02 14:55:49.952281 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.952260    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swqf5" event={"ID":"8ca09ed6-20a8-4b83-bae6-a371cbf71d65","Type":"ContainerStarted","Data":"0470e194ffd5f56ac2a59943815e04c4df0089015b2c45c1ce827e6a0fae3493"}
Nov 02 14:55:49.953448 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.953426    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th" event={"ID":"86f99765-b728-464e-bce3-fb178f34d0ed","Type":"ContainerStarted","Data":"e84043694b2cf8e94263d1bee1a6d6ed1ed35a8da2b53d69a83b6bc88957153a"}
Nov 02 14:55:49.954526 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:49.954504    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfqf9" event={"ID":"a3be13e2-c47a-4a45-b023-fff5e1ec32ee","Type":"ContainerStarted","Data":"a03d1f9a1f3b1521f4e9d32a61ecdcf00fa384ba83a7865b473bfc766190959c"}
Nov 02 14:55:50.089912 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.089802    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f8679cb6-t6942"]
Nov 02 14:55:50.095662 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:50.095632    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f276b8e_7e4f_4980_a8a3_9fed6ee92685.slice/crio-07a6c27ea1147a00d429f97b293087ed2219c54143870b63c15e986899fea80c WatchSource:0}: Error finding container 07a6c27ea1147a00d429f97b293087ed2219c54143870b63c15e986899fea80c: Status 404 returned error can't find the container with id 07a6c27ea1147a00d429f97b293087ed2219c54143870b63c15e986899fea80c
Nov 02 14:55:50.096393 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.096352    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588df7d7d7-sv5v5"]
Nov 02 14:55:50.098573 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:50.096807    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e33558_be4f_4b8b_812e_f73322516ac0.slice/crio-eeea2430725f0f107ecdd3c961d2f06ccdd6afc3612be52b67ad0ea0a22b283d WatchSource:0}: Error finding container eeea2430725f0f107ecdd3c961d2f06ccdd6afc3612be52b67ad0ea0a22b283d: Status 404 returned error can't find the container with id eeea2430725f0f107ecdd3c961d2f06ccdd6afc3612be52b67ad0ea0a22b283d
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.621039    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.621084    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.621105    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.621132    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.621169    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621307    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.621294579 +0000 UTC m=+4.879364006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : configmap references non-existent config key: service-ca.crt
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621612    5041 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621649    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.621639211 +0000 UTC m=+4.879708639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621700    5041 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621720    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs podName:e6beba97-264e-4107-bafa-57b5380ad218 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.621713875 +0000 UTC m=+4.879783304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs") pod "router-default-659675459f-cp5t4" (UID: "e6beba97-264e-4107-bafa-57b5380ad218") : secret "router-metrics-certs-default" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621769    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621792    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.621785883 +0000 UTC m=+4.879855311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621830    5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:50.622216 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.621861    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.621851923 +0000 UTC m=+4.879921351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:50.967613 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.966662    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5" event={"ID":"58e33558-be4f-4b8b-812e-f73322516ac0","Type":"ContainerStarted","Data":"c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d"}
Nov 02 14:55:50.967613 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.966722    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5" event={"ID":"58e33558-be4f-4b8b-812e-f73322516ac0","Type":"ContainerStarted","Data":"eeea2430725f0f107ecdd3c961d2f06ccdd6afc3612be52b67ad0ea0a22b283d"}
Nov 02 14:55:50.967613 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.967567    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:55:50.971660 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.970660    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 14:55:50.971660 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.971040    5041 scope.go:122] "RemoveContainer" containerID="92071144c8b133f87b71ef086c56c6899ac0b5167d2edc9d86941a32dbc1f610"
Nov 02 14:55:50.971660 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:50.971226    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlsqw_openshift-console-operator(113ca502-9379-4a88-b3e7-3599fe9cbfd9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" podUID="113ca502-9379-4a88-b3e7-3599fe9cbfd9"
Nov 02 14:55:50.975078 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.974690    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" event={"ID":"0f276b8e-7e4f-4980-a8a3-9fed6ee92685","Type":"ContainerStarted","Data":"6966800081a2b11f9d38389930628bbef2c922084d08146a06656b94e10be85f"}
Nov 02 14:55:50.975078 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:50.974728    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" event={"ID":"0f276b8e-7e4f-4980-a8a3-9fed6ee92685","Type":"ContainerStarted","Data":"07a6c27ea1147a00d429f97b293087ed2219c54143870b63c15e986899fea80c"}
Nov 02 14:55:51.981960 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.981927    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cfnwj" event={"ID":"bf141b7d-a42e-46d1-b71b-987f2c080604","Type":"ContainerStarted","Data":"3b890e4552d4d08a9dabc6085841c6dbc6c30793d1d8ca0dcbe0ade3843cb1a6"}
Nov 02 14:55:51.983942 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.983915    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swqf5" event={"ID":"8ca09ed6-20a8-4b83-bae6-a371cbf71d65","Type":"ContainerStarted","Data":"9dc1a1351b04a2071126c6c64e484bc66369dab6e42b35101ed24d067c140a88"}
Nov 02 14:55:51.984050 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.983953    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swqf5" event={"ID":"8ca09ed6-20a8-4b83-bae6-a371cbf71d65","Type":"ContainerStarted","Data":"80a2901992753d4fab192d5f8ff7e1ee01d8ce7dcf4102c1c720b3d675f3ef2c"}
Nov 02 14:55:51.984050 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.983982    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-swqf5"
Nov 02 14:55:51.985666 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.985648    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th" event={"ID":"86f99765-b728-464e-bce3-fb178f34d0ed","Type":"ContainerStarted","Data":"f67c466fb2b6f864b7c60d72bece9f5e24cc93dce732d7cde3f334af9ead360d"}
Nov 02 14:55:51.985750 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.985689    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l66th" event={"ID":"86f99765-b728-464e-bce3-fb178f34d0ed","Type":"ContainerStarted","Data":"81175d5d1ddbbce0ff2ed84fa0fcfadc45b047fa904e5e5f4d0591adee478f8b"}
Nov 02 14:55:51.987636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.987616    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfqf9" event={"ID":"a3be13e2-c47a-4a45-b023-fff5e1ec32ee","Type":"ContainerStarted","Data":"5d328151ebfa92b2fcfb145673905f40a05dc4df661ff42a7da1499146dd29a7"}
Nov 02 14:55:51.987704 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.987650    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:55:51.987704 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:51.987657    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfqf9" event={"ID":"a3be13e2-c47a-4a45-b023-fff5e1ec32ee","Type":"ContainerStarted","Data":"374638c2d8e1759a4f7a185a5ff7aaafeacc58ab8f49b5976b18fd9a704e2ab4"}
Nov 02 14:55:52.643988 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.643933    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:52.643988 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.643982    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.644002    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.644020    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.644047    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:52.644065    5041 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:52.644121    5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:52.644144    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:56.644124599 +0000 UTC m=+8.902194027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : configmap "kube-rbac-proxy" not found
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:52.644152    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:52.644183 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:52.644171    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls podName:eb5081a6-d3c8-4971-9e46-49f330c7913f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:56.64416071 +0000 UTC m=+8.902230138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8dg29" (UID: "eb5081a6-d3c8-4971-9e46-49f330c7913f") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:55:52.644564 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:52.644200    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls podName:e5150492-44d2-426a-a833-69856c7ddf09 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:56.644190078 +0000 UTC m=+8.902259527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzg46" (UID: "e5150492-44d2-426a-a833-69856c7ddf09") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:55:52.644636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.644618    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6beba97-264e-4107-bafa-57b5380ad218-service-ca-bundle\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:52.646229 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.646213    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6beba97-264e-4107-bafa-57b5380ad218-metrics-certs\") pod \"router-default-659675459f-cp5t4\" (UID: \"e6beba97-264e-4107-bafa-57b5380ad218\") " pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:52.782287 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.782264    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:52.991685 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.991657    5041 generic.go:356] "Generic (PLEG): container finished" podID="40c08878-17e0-4207-8b21-a3e3da129d37" containerID="e65bc4955f78d08008d5c1c358ea1b6bbdc4d9b05c0bbf8cecde741fc3c25455" exitCode=0
Nov 02 14:55:52.992069 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.991730    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" event={"ID":"40c08878-17e0-4207-8b21-a3e3da129d37","Type":"ContainerDied","Data":"e65bc4955f78d08008d5c1c358ea1b6bbdc4d9b05c0bbf8cecde741fc3c25455"}
Nov 02 14:55:52.992126 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.992111    5041 scope.go:122] "RemoveContainer" containerID="e65bc4955f78d08008d5c1c358ea1b6bbdc4d9b05c0bbf8cecde741fc3c25455"
Nov 02 14:55:52.993264 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.993237    5041 generic.go:356] "Generic (PLEG): container finished" podID="dc383de8-7d40-46ee-84ff-74b20040ff2c" containerID="e9cbca2cbe50f921880e96cddfe3f2c7d8850e30796bc91179f4c4faeba206f3" exitCode=0
Nov 02 14:55:52.993373 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.993261    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz" event={"ID":"dc383de8-7d40-46ee-84ff-74b20040ff2c","Type":"ContainerDied","Data":"e9cbca2cbe50f921880e96cddfe3f2c7d8850e30796bc91179f4c4faeba206f3"}
Nov 02 14:55:52.993540 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.993526    5041 scope.go:122] "RemoveContainer" containerID="e9cbca2cbe50f921880e96cddfe3f2c7d8850e30796bc91179f4c4faeba206f3"
Nov 02 14:55:52.999144 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.999123    5041 generic.go:356] "Generic (PLEG): container finished" podID="6831662e-48ae-42bc-95b4-27633e745504" containerID="96b13d8a18eefa16a61cbbb32a0f3b673aefc41a53c2dac2f225228cdd0bc379" exitCode=0
Nov 02 14:55:52.999258 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.999238    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42" event={"ID":"6831662e-48ae-42bc-95b4-27633e745504","Type":"ContainerDied","Data":"96b13d8a18eefa16a61cbbb32a0f3b673aefc41a53c2dac2f225228cdd0bc379"}
Nov 02 14:55:52.999713 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:52.999698    5041 scope.go:122] "RemoveContainer" containerID="96b13d8a18eefa16a61cbbb32a0f3b673aefc41a53c2dac2f225228cdd0bc379"
Nov 02 14:55:53.120073 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:53.120034    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-659675459f-cp5t4"]
Nov 02 14:55:53.123893 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:53.123869    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6beba97_264e_4107_bafa_57b5380ad218.slice/crio-32a7ef5a0f84511121a303e482da8dadfe905360cfdee64dae74879f3c148c06 WatchSource:0}: Error finding container 32a7ef5a0f84511121a303e482da8dadfe905360cfdee64dae74879f3c148c06: Status 404 returned error can't find the container with id 32a7ef5a0f84511121a303e482da8dadfe905360cfdee64dae74879f3c148c06
Nov 02 14:55:54.005631 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.005594    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-42nrz" event={"ID":"dc383de8-7d40-46ee-84ff-74b20040ff2c","Type":"ContainerStarted","Data":"c61ffc43bb38653920fc9b390a1227b2784a689a385c1e9dc1d4597b0a568324"}
Nov 02 14:55:54.007144 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.007120    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-659675459f-cp5t4" event={"ID":"e6beba97-264e-4107-bafa-57b5380ad218","Type":"ContainerStarted","Data":"ae35a5c49f2ffb94bf151b241850ef68b7d2b5ef19fcb12bbc7cbebca043afda"}
Nov 02 14:55:54.007256 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.007155    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-659675459f-cp5t4" event={"ID":"e6beba97-264e-4107-bafa-57b5380ad218","Type":"ContainerStarted","Data":"32a7ef5a0f84511121a303e482da8dadfe905360cfdee64dae74879f3c148c06"}
Nov 02 14:55:54.008999 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.008975    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v5k42" event={"ID":"6831662e-48ae-42bc-95b4-27633e745504","Type":"ContainerStarted","Data":"d5e36ed67437f35aaf98793b0d11f782b01b2e8047ad63bc6df10b04e9651071"}
Nov 02 14:55:54.010777 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.010757    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" event={"ID":"40c08878-17e0-4207-8b21-a3e3da129d37","Type":"ContainerStarted","Data":"70a3ef34650e9943197937201717c921ffbb7adccf86455396bef6d0bb0bb111"}
Nov 02 14:55:54.116748 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.116725    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:55:54.116949 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.116934    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:55:54.782499 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.782461    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:54.785113 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:54.785088    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:55.988814 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:55.988781    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:55.989192 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:55.989130    5041 scope.go:122] "RemoveContainer" containerID="92071144c8b133f87b71ef086c56c6899ac0b5167d2edc9d86941a32dbc1f610"
Nov 02 14:55:55.989317 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:55.989296    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlsqw_openshift-console-operator(113ca502-9379-4a88-b3e7-3599fe9cbfd9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" podUID="113ca502-9379-4a88-b3e7-3599fe9cbfd9"
Nov 02 14:55:56.019106 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.019079    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:55:56.021440 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.021425    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:55:56.681873 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.681834    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:56.682054 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.681884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:56.682054 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.681924    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:56.682487 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.682468    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5150492-44d2-426a-a833-69856c7ddf09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:56.684276 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.684247    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5150492-44d2-426a-a833-69856c7ddf09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzg46\" (UID: \"e5150492-44d2-426a-a833-69856c7ddf09\") " pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:56.684383 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.684356    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb5081a6-d3c8-4971-9e46-49f330c7913f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8dg29\" (UID: \"eb5081a6-d3c8-4971-9e46-49f330c7913f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:56.694249 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.694230    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzg46"
Nov 02 14:55:56.981587 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.981514    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"
Nov 02 14:55:56.986927 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:56.986898    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzg46"]
Nov 02 14:55:56.990174 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:56.990146    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5150492_44d2_426a_a833_69856c7ddf09.slice/crio-8dd18ddf963e43af8ba49e295fd0587072290d22d41ca12333e27bb2cfa5f73a WatchSource:0}: Error finding container 8dd18ddf963e43af8ba49e295fd0587072290d22d41ca12333e27bb2cfa5f73a: Status 404 returned error can't find the container with id 8dd18ddf963e43af8ba49e295fd0587072290d22d41ca12333e27bb2cfa5f73a
Nov 02 14:55:57.026565 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:57.026538    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzg46" event={"ID":"e5150492-44d2-426a-a833-69856c7ddf09","Type":"ContainerStarted","Data":"8dd18ddf963e43af8ba49e295fd0587072290d22d41ca12333e27bb2cfa5f73a"}
Nov 02 14:55:57.277785 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:57.277712    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29"]
Nov 02 14:55:57.281501 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:55:57.281477    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5081a6_d3c8_4971_9e46_49f330c7913f.slice/crio-f6d613eb66f103e42228cd10d4d234e99fe6dba17fee765f0fa1589c144e2cb1 WatchSource:0}: Error finding container f6d613eb66f103e42228cd10d4d234e99fe6dba17fee765f0fa1589c144e2cb1: Status 404 returned error can't find the container with id f6d613eb66f103e42228cd10d4d234e99fe6dba17fee765f0fa1589c144e2cb1
Nov 02 14:55:58.032872 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.032759    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzg46" event={"ID":"e5150492-44d2-426a-a833-69856c7ddf09","Type":"ContainerStarted","Data":"53b811c896074470e000c5cbc194f42cd97420cc7f5ac272f0292b0beeeda004"}
Nov 02 14:55:58.032872 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.032833    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzg46" event={"ID":"e5150492-44d2-426a-a833-69856c7ddf09","Type":"ContainerStarted","Data":"6249248c3da24779a3a1fc1439ee4cd7e790a8fa5ce0fbc54127fc0bbe013876"}
Nov 02 14:55:58.034291 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.034250    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29" event={"ID":"eb5081a6-d3c8-4971-9e46-49f330c7913f","Type":"ContainerStarted","Data":"f6d613eb66f103e42228cd10d4d234e99fe6dba17fee765f0fa1589c144e2cb1"}
Nov 02 14:55:58.247906 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.247871    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:55:58.525879 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.525837    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:58.526048 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.525994    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:55:58.527280 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:58.527257    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-659675459f-cp5t4"
Nov 02 14:55:59.040278 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.040247    5041 generic.go:356] "Generic (PLEG): container finished" podID="40c08878-17e0-4207-8b21-a3e3da129d37" containerID="70a3ef34650e9943197937201717c921ffbb7adccf86455396bef6d0bb0bb111" exitCode=0
Nov 02 14:55:59.040643 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.040317    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" event={"ID":"40c08878-17e0-4207-8b21-a3e3da129d37","Type":"ContainerDied","Data":"70a3ef34650e9943197937201717c921ffbb7adccf86455396bef6d0bb0bb111"}
Nov 02 14:55:59.040643 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.040395    5041 scope.go:122] "RemoveContainer" containerID="e65bc4955f78d08008d5c1c358ea1b6bbdc4d9b05c0bbf8cecde741fc3c25455"
Nov 02 14:55:59.040754 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.040698    5041 scope.go:122] "RemoveContainer" containerID="70a3ef34650e9943197937201717c921ffbb7adccf86455396bef6d0bb0bb111"
Nov 02 14:55:59.040937 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:55:59.040917    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-8nlz6_openshift-insights(40c08878-17e0-4207-8b21-a3e3da129d37)\"" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" podUID="40c08878-17e0-4207-8b21-a3e3da129d37"
Nov 02 14:55:59.041932 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.041905    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8dg29" event={"ID":"eb5081a6-d3c8-4971-9e46-49f330c7913f","Type":"ContainerStarted","Data":"68d37fb61fb22ab5b93e06baafb1084e9693b20d3967446c6e2df4f92cb1aacb"}
Nov 02 14:55:59.666666 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.666631    5041 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:55:59.667003 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:55:59.666990    5041 scope.go:122] "RemoveContainer" containerID="92071144c8b133f87b71ef086c56c6899ac0b5167d2edc9d86941a32dbc1f610"
Nov 02 14:56:00.047857 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:00.047825    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 14:56:00.048322 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:00.047925    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw" event={"ID":"113ca502-9379-4a88-b3e7-3599fe9cbfd9","Type":"ContainerStarted","Data":"e02d09baf2868dc7eb74e5e95d0401ddd508bd9705ff9edaf323e1696e6b5456"}
Nov 02 14:56:00.048322 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:00.048268    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:56:00.056744 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:00.056707    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzg46" event={"ID":"e5150492-44d2-426a-a833-69856c7ddf09","Type":"ContainerStarted","Data":"564f4454af29ba9bb50f28f54fbcf76a8127892c7d0360a855af7cf0a348daa9"}
Nov 02 14:56:00.185621 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:00.185590    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlsqw"
Nov 02 14:56:02.001760 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:02.001727    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-swqf5"
Nov 02 14:56:06.027156 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:06.027119    5041 patch_prober.go:29] interesting pod/image-registry-84f8679cb6-t6942 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:56:06.027561 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:06.027174    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:56:06.397116 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:06.397032    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-svzrl"
Nov 02 14:56:06.767387 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:56:06.767338    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:56:07.139073 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.138994    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-28f2f-m4r8k"]
Nov 02 14:56:07.143534 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.143509    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.191638 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.191606    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-588df7d7d7-sv5v5"]
Nov 02 14:56:07.196956 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.196919    5041 patch_prober.go:29] interesting pod/image-registry-588df7d7d7-sv5v5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:56:07.197103 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.196972    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5" podUID="58e33558-be4f-4b8b-812e-f73322516ac0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:56:07.204261 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.204233    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-r2nzt"]
Nov 02 14:56:07.208133 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.208109    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"]
Nov 02 14:56:07.208258 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.208240    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.212597 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.212577    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:07.239475 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.239454    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 14:56:07.239601 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.239517    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zjw27\""
Nov 02 14:56:07.239851 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.239834    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 14:56:07.239851 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.239836    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9jmb4\""
Nov 02 14:56:07.240708 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.240694    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 14:56:07.243181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.243161    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"]
Nov 02 14:56:07.243904 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.243888    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-r2nzt"]
Nov 02 14:56:07.260329 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.260310    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-28f2f-m4r8k"]
Nov 02 14:56:07.271639 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.271619    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.271748 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.271650    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/59843d2a-5c16-4caf-9c1e-bdf97c44401f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fmjcg\" (UID: \"59843d2a-5c16-4caf-9c1e-bdf97c44401f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:07.271748 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.271668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgvr\" (UniqueName: \"kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.271846 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.271809    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.271893 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.271842    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.286932 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.286911    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b46b95748-pwxs9"]
Nov 02 14:56:07.290116 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.290100    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.317267 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.317241    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b46b95748-pwxs9"]
Nov 02 14:56:07.373132 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373106    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.373257 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373139    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/59843d2a-5c16-4caf-9c1e-bdf97c44401f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fmjcg\" (UID: \"59843d2a-5c16-4caf-9c1e-bdf97c44401f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:07.373257 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373159    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgvr\" (UniqueName: \"kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.373257 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373179    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-image-registry-private-configuration\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.373257 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.373257 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.373584 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373383    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-registry-certificates\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.373584 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373423    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-installation-pull-secrets\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.373584 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373468    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41a77928-03ff-4526-9e95-10c7ae70657f-tmp\") pod \"downloads-6dd789c8b7-r2nzt\" (UID: \"41a77928-03ff-4526-9e95-10c7ae70657f\") " pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.373584 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373503    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-registry-tls\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.373792 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373606    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.373792 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373623    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-ca-trust-extracted\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.373792 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqqf\" (UniqueName: \"kubernetes.io/projected/41a77928-03ff-4526-9e95-10c7ae70657f-kube-api-access-2gqqf\") pod \"downloads-6dd789c8b7-r2nzt\" (UID: \"41a77928-03ff-4526-9e95-10c7ae70657f\") " pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.373792 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373736    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-trusted-ca\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.373792 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373763    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg42n\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-kube-api-access-rg42n\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.374029 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373846    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-bound-sa-token\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.374029 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.373951    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.374148 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.374129    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.375555 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.375539    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/59843d2a-5c16-4caf-9c1e-bdf97c44401f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fmjcg\" (UID: \"59843d2a-5c16-4caf-9c1e-bdf97c44401f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:07.401778 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.401724    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgvr\" (UniqueName: \"kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr\") pod \"periodic-gathering-28f2f-m4r8k\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") " pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.457780 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.457760    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:07.474878 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.474855    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-bound-sa-token\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.474974 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.474912    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-image-registry-private-configuration\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.474974 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.474959    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-registry-certificates\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475083 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.474986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-installation-pull-secrets\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475083 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475024    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41a77928-03ff-4526-9e95-10c7ae70657f-tmp\") pod \"downloads-6dd789c8b7-r2nzt\" (UID: \"41a77928-03ff-4526-9e95-10c7ae70657f\") " pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.475083 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475048    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-registry-tls\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475083 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475075    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-ca-trust-extracted\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475293 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475114    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqqf\" (UniqueName: \"kubernetes.io/projected/41a77928-03ff-4526-9e95-10c7ae70657f-kube-api-access-2gqqf\") pod \"downloads-6dd789c8b7-r2nzt\" (UID: \"41a77928-03ff-4526-9e95-10c7ae70657f\") " pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.475293 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475137    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-trusted-ca\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475293 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475174    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg42n\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-kube-api-access-rg42n\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475648 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475625    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41a77928-03ff-4526-9e95-10c7ae70657f-tmp\") pod \"downloads-6dd789c8b7-r2nzt\" (UID: \"41a77928-03ff-4526-9e95-10c7ae70657f\") " pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.475773 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475752    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-ca-trust-extracted\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.475890 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.475867    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-registry-certificates\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.476561 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.476537    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-trusted-ca\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.477424 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.477394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-registry-tls\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.477669 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.477647    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-installation-pull-secrets\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.477938 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.477923    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-image-registry-private-configuration\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.499278 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.499251    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqqf\" (UniqueName: \"kubernetes.io/projected/41a77928-03ff-4526-9e95-10c7ae70657f-kube-api-access-2gqqf\") pod \"downloads-6dd789c8b7-r2nzt\" (UID: \"41a77928-03ff-4526-9e95-10c7ae70657f\") " pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.499397 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.499280    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-bound-sa-token\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.499520 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.499496    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg42n\" (UniqueName: \"kubernetes.io/projected/2c84d995-8e44-41e0-95c4-1bd7f16f64fc-kube-api-access-rg42n\") pod \"image-registry-7b46b95748-pwxs9\" (UID: \"2c84d995-8e44-41e0-95c4-1bd7f16f64fc\") " pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.521783 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.521758    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:07.527204 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.526999    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:07.604404 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.604380    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:07.750449 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.750415    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-28f2f-m4r8k"]
Nov 02 14:56:07.753154 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:07.753131    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bc2365_fcbd_4dd1_9fda_bcad3e390603.slice/crio-81526780d50549142a5489c75317e4a90d43ec7272f2b2a4441e9beee48d74dd WatchSource:0}: Error finding container 81526780d50549142a5489c75317e4a90d43ec7272f2b2a4441e9beee48d74dd: Status 404 returned error can't find the container with id 81526780d50549142a5489c75317e4a90d43ec7272f2b2a4441e9beee48d74dd
Nov 02 14:56:07.839210 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.839185    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-r2nzt"]
Nov 02 14:56:07.841681 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:07.841645    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a77928_03ff_4526_9e95_10c7ae70657f.slice/crio-bab2bbdbfd9d883591f13095f7c204f4b9c3857143bb22fcd4211c82f6027c5a WatchSource:0}: Error finding container bab2bbdbfd9d883591f13095f7c204f4b9c3857143bb22fcd4211c82f6027c5a: Status 404 returned error can't find the container with id bab2bbdbfd9d883591f13095f7c204f4b9c3857143bb22fcd4211c82f6027c5a
Nov 02 14:56:07.841681 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.841652    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"]
Nov 02 14:56:07.844975 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:07.844914    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59843d2a_5c16_4caf_9c1e_bdf97c44401f.slice/crio-d04ee0e7c6724541745d19b877dd058a475d57cfdd4c634771070d9cbc377245 WatchSource:0}: Error finding container d04ee0e7c6724541745d19b877dd058a475d57cfdd4c634771070d9cbc377245: Status 404 returned error can't find the container with id d04ee0e7c6724541745d19b877dd058a475d57cfdd4c634771070d9cbc377245
Nov 02 14:56:07.883919 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.883889    5041 patch_prober.go:29] interesting pod/image-registry-84f8679cb6-t6942 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:56:07.884057 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.883937    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:56:07.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:07.905860    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b46b95748-pwxs9"]
Nov 02 14:56:07.908876 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:07.908855    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c84d995_8e44_41e0_95c4_1bd7f16f64fc.slice/crio-f5171d60cbc3241b776f21ef30e3a0cdf4b66e878dd7405e0ef44151ebf88874 WatchSource:0}: Error finding container f5171d60cbc3241b776f21ef30e3a0cdf4b66e878dd7405e0ef44151ebf88874: Status 404 returned error can't find the container with id f5171d60cbc3241b776f21ef30e3a0cdf4b66e878dd7405e0ef44151ebf88874
Nov 02 14:56:08.088785 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.088743    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-r2nzt" event={"ID":"41a77928-03ff-4526-9e95-10c7ae70657f","Type":"ContainerStarted","Data":"bab2bbdbfd9d883591f13095f7c204f4b9c3857143bb22fcd4211c82f6027c5a"}
Nov 02 14:56:08.090059 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.090029    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg" event={"ID":"59843d2a-5c16-4caf-9c1e-bdf97c44401f","Type":"ContainerStarted","Data":"d04ee0e7c6724541745d19b877dd058a475d57cfdd4c634771070d9cbc377245"}
Nov 02 14:56:08.091652 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.091625    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9" event={"ID":"2c84d995-8e44-41e0-95c4-1bd7f16f64fc","Type":"ContainerStarted","Data":"96e824f0cf05010c435a455b6976e68b607f976105b7c30e557be3bb1cfcaf6f"}
Nov 02 14:56:08.091761 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.091672    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9" event={"ID":"2c84d995-8e44-41e0-95c4-1bd7f16f64fc","Type":"ContainerStarted","Data":"f5171d60cbc3241b776f21ef30e3a0cdf4b66e878dd7405e0ef44151ebf88874"}
Nov 02 14:56:08.091761 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.091726    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:08.093157 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.093136    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-28f2f-m4r8k" event={"ID":"e1bc2365-fcbd-4dd1-9fda-bcad3e390603","Type":"ContainerStarted","Data":"95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883"}
Nov 02 14:56:08.093267 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.093171    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-28f2f-m4r8k" event={"ID":"e1bc2365-fcbd-4dd1-9fda-bcad3e390603","Type":"ContainerStarted","Data":"81526780d50549142a5489c75317e4a90d43ec7272f2b2a4441e9beee48d74dd"}
Nov 02 14:56:08.155654 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.155603    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9" podStartSLOduration=1.155590318 podStartE2EDuration="1.155590318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:08.154535333 +0000 UTC m=+20.412604800" watchObservedRunningTime="2026-11-02 14:56:08.155590318 +0000 UTC m=+20.413659765"
Nov 02 14:56:08.231062 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.230995    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-28f2f-m4r8k" podStartSLOduration=1.230975772 podStartE2EDuration="1.230975772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:08.229392137 +0000 UTC m=+20.487461581" watchObservedRunningTime="2026-11-02 14:56:08.230975772 +0000 UTC m=+20.489045220"
Nov 02 14:56:08.846971 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:08.846880    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swqf5_8ca09ed6-20a8-4b83-bae6-a371cbf71d65/dns/0.log"
Nov 02 14:56:09.047062 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.047034    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swqf5_8ca09ed6-20a8-4b83-bae6-a371cbf71d65/kube-rbac-proxy/0.log"
Nov 02 14:56:09.099004 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.098914    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg" event={"ID":"59843d2a-5c16-4caf-9c1e-bdf97c44401f","Type":"ContainerStarted","Data":"881542b28e6231087d0e317fe532c86d8799fb10bec4c351cddf8de853bdf4e9"}
Nov 02 14:56:09.099398 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.099379    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:09.105350 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.105326    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg"
Nov 02 14:56:09.224914 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.224854    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fmjcg" podStartSLOduration=1.180568009 podStartE2EDuration="2.224837765s" totalImagesPullingTime="1.044269756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:09.185257997 +0000 UTC m=+21.443327448" watchObservedRunningTime="2026-11-02 14:56:09.224837765 +0000 UTC m=+21.482907209"
Nov 02 14:56:09.249512 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.249483    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g4d2h_d71694a7-787c-45ec-9308-6ff029eead3f/dns-node-resolver/0.log"
Nov 02 14:56:09.847681 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:09.847644    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-588df7d7d7-sv5v5_58e33558-be4f-4b8b-812e-f73322516ac0/registry/0.log"
Nov 02 14:56:10.047769 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.047734    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b46b95748-pwxs9_2c84d995-8e44-41e0-95c4-1bd7f16f64fc/registry/0.log"
Nov 02 14:56:10.051816 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.050329    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"]
Nov 02 14:56:10.078433 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.078348    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"]
Nov 02 14:56:10.078593 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.078472    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.092701 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.092663    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 14:56:10.101553 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.101443    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42a79555-3053-449e-b985-022a5a5a7fb5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.101553 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.101491    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxjf\" (UniqueName: \"kubernetes.io/projected/42a79555-3053-449e-b985-022a5a5a7fb5-kube-api-access-mbxjf\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.101739 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.101605    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42a79555-3053-449e-b985-022a5a5a7fb5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.101739 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.101638    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42a79555-3053-449e-b985-022a5a5a7fb5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.104522 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.104493    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r2spk\""
Nov 02 14:56:10.105317 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.105298    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 14:56:10.105715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.105695    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 14:56:10.202401 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.202338    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxjf\" (UniqueName: \"kubernetes.io/projected/42a79555-3053-449e-b985-022a5a5a7fb5-kube-api-access-mbxjf\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.202575 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.202424    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42a79555-3053-449e-b985-022a5a5a7fb5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.202575 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.202443    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42a79555-3053-449e-b985-022a5a5a7fb5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.202705 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.202535    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42a79555-3053-449e-b985-022a5a5a7fb5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.203447 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.203427    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42a79555-3053-449e-b985-022a5a5a7fb5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.205130 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.205106    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42a79555-3053-449e-b985-022a5a5a7fb5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.205336 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.205320    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42a79555-3053-449e-b985-022a5a5a7fb5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.225651 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.225627    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxjf\" (UniqueName: \"kubernetes.io/projected/42a79555-3053-449e-b985-022a5a5a7fb5-kube-api-access-mbxjf\") pod \"prometheus-operator-9cd89b74c-n9v7w\" (UID: \"42a79555-3053-449e-b985-022a5a5a7fb5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.247063 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.247039    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84f8679cb6-t6942_0f276b8e-7e4f-4980-a8a3-9fed6ee92685/registry/0.log"
Nov 02 14:56:10.427358 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.427327    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"
Nov 02 14:56:10.749885 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.749794    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w"]
Nov 02 14:56:10.752316 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:10.752289    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a79555_3053_449e_b985_022a5a5a7fb5.slice/crio-d388079dd04e25d30cb7732199c9fbfc4149a6fb21022b48c48234f1040178e4 WatchSource:0}: Error finding container d388079dd04e25d30cb7732199c9fbfc4149a6fb21022b48c48234f1040178e4: Status 404 returned error can't find the container with id d388079dd04e25d30cb7732199c9fbfc4149a6fb21022b48c48234f1040178e4
Nov 02 14:56:10.847627 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.847604    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm8rc_ba225e36-3f26-42cf-b829-4e3117f7e899/node-ca/0.log"
Nov 02 14:56:10.880256 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:10.880227    5041 scope.go:122] "RemoveContainer" containerID="70a3ef34650e9943197937201717c921ffbb7adccf86455396bef6d0bb0bb111"
Nov 02 14:56:11.048335 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:11.048226    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659675459f-cp5t4_e6beba97-264e-4107-bafa-57b5380ad218/router/0.log"
Nov 02 14:56:11.113432 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:11.113394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w" event={"ID":"42a79555-3053-449e-b985-022a5a5a7fb5","Type":"ContainerStarted","Data":"d388079dd04e25d30cb7732199c9fbfc4149a6fb21022b48c48234f1040178e4"}
Nov 02 14:56:11.116025 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:11.115546    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8nlz6" event={"ID":"40c08878-17e0-4207-8b21-a3e3da129d37","Type":"ContainerStarted","Data":"8df5d3766dbf82f71d92fdea36648f401fa3bf54a80acb48f0667dac58a8caaa"}
Nov 02 14:56:11.257231 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:11.257195    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8jgh_c9af00a4-820f-473f-bc2d-7141fbe02975/serve-healthcheck-canary/0.log"
Nov 02 14:56:12.334706 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.334667    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7585cfc69f-nwnpt"]
Nov 02 14:56:12.359307 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.359281    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.361521 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.361490    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7585cfc69f-nwnpt"]
Nov 02 14:56:12.384756 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.384674    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 14:56:12.384756 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.384741    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 14:56:12.385004 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.384892    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 14:56:12.385004 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.384932    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 14:56:12.385004 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.384984    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6c5wx\""
Nov 02 14:56:12.385004 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.384995    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 14:56:12.428100 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428079    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.428194 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428112    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.428194 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428149    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.428292 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428247    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.428341 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428295    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4w9z\" (UniqueName: \"kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.428414 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428389    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.428510 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.428493    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529003 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.528968    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529045    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529084    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529105    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529127 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529127    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529301 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529144    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529301 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529175    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4w9z\" (UniqueName: \"kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.529923 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.529903    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.530121 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.530101    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.530732 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.530690    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.531064 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.531039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.544395 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.534805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.544395 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.534891    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.566454 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.566422    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4w9z\" (UniqueName: \"kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z\") pod \"console-7585cfc69f-nwnpt\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") " pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:12.692443 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:12.692405    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:13.011023 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:13.010918    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7585cfc69f-nwnpt"]
Nov 02 14:56:13.014261 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:13.014225    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7680b9c1_602a_4a6a_89e1_e90b8f77d980.slice/crio-6e6c43a8590c0219659bc2f0f3da39c4c296f95cd53bee80f9e410e54bc7ffe1 WatchSource:0}: Error finding container 6e6c43a8590c0219659bc2f0f3da39c4c296f95cd53bee80f9e410e54bc7ffe1: Status 404 returned error can't find the container with id 6e6c43a8590c0219659bc2f0f3da39c4c296f95cd53bee80f9e410e54bc7ffe1
Nov 02 14:56:13.126754 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:13.126712    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585cfc69f-nwnpt" event={"ID":"7680b9c1-602a-4a6a-89e1-e90b8f77d980","Type":"ContainerStarted","Data":"6e6c43a8590c0219659bc2f0f3da39c4c296f95cd53bee80f9e410e54bc7ffe1"}
Nov 02 14:56:13.128837 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:13.128807    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w" event={"ID":"42a79555-3053-449e-b985-022a5a5a7fb5","Type":"ContainerStarted","Data":"fda7b5675409943c6c5fc1eb0fc89eb2c5574004906a698a2529e1ce3de2f75f"}
Nov 02 14:56:13.128983 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:13.128859    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w" event={"ID":"42a79555-3053-449e-b985-022a5a5a7fb5","Type":"ContainerStarted","Data":"f97de020efeaf7e0dc3cef5fcb161f3bc77f2ec9d451b0abefdbaaedb31e277e"}
Nov 02 14:56:13.169402 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:13.169325    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n9v7w" podStartSLOduration=1.511993412 podStartE2EDuration="3.169306558s" totalImagesPullingTime="1.657313146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:13.169035619 +0000 UTC m=+25.427105070" watchObservedRunningTime="2026-11-02 14:56:13.169306558 +0000 UTC m=+25.427376010"
Nov 02 14:56:15.441508 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.440736    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjtlf"
Nov 02 14:56:15.735546 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.735465    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5hfx8"]
Nov 02 14:56:15.790826 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.790793    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.853772 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.853614    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 14:56:15.853772 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.853738    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 14:56:15.854325 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.853942    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nts94\""
Nov 02 14:56:15.854420 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.854178    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 14:56:15.864667 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864639    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-textfile\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.864786 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864687    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-wtmp\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.864786 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864715    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.864786 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864778    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-tls\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.864954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864829    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.864954 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864940    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-root\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.865018 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.864976    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8287a73-bfc6-492d-8be0-109ba812e3af-metrics-client-ca\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.865018 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.865014    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vpk\" (UniqueName: \"kubernetes.io/projected/c8287a73-bfc6-492d-8be0-109ba812e3af-kube-api-access-56vpk\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.865079 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.865046    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-sys\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.965857 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.965820    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-textfile\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.965857 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.965872    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-wtmp\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966102 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.965999    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966102 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966017    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-wtmp\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966102 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966048    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-tls\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966102 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966080    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966223 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966110    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-root\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966223 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966149    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8287a73-bfc6-492d-8be0-109ba812e3af-metrics-client-ca\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966223 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966201    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vpk\" (UniqueName: \"kubernetes.io/projected/c8287a73-bfc6-492d-8be0-109ba812e3af-kube-api-access-56vpk\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966223 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966212    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-textfile\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.966411 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966284    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-root\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.967068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966797    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.967068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966826    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8287a73-bfc6-492d-8be0-109ba812e3af-metrics-client-ca\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.967068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966834    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-sys\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.967068 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.966984    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8287a73-bfc6-492d-8be0-109ba812e3af-sys\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.968916 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.968877    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-tls\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.969016 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.968925    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8287a73-bfc6-492d-8be0-109ba812e3af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:15.991915 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:15.991840    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vpk\" (UniqueName: \"kubernetes.io/projected/c8287a73-bfc6-492d-8be0-109ba812e3af-kube-api-access-56vpk\") pod \"node-exporter-5hfx8\" (UID: \"c8287a73-bfc6-492d-8be0-109ba812e3af\") " pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:16.030412 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:16.030174    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:56:16.202469 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:16.202433    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5hfx8"
Nov 02 14:56:16.280769 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:16.280733    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8287a73_bfc6_492d_8be0_109ba812e3af.slice/crio-b074541fd9ba8e3764ae411d0fb267d26880039230bbf8e6a71efb26fc32da78 WatchSource:0}: Error finding container b074541fd9ba8e3764ae411d0fb267d26880039230bbf8e6a71efb26fc32da78: Status 404 returned error can't find the container with id b074541fd9ba8e3764ae411d0fb267d26880039230bbf8e6a71efb26fc32da78
Nov 02 14:56:17.152353 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:17.152279    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585cfc69f-nwnpt" event={"ID":"7680b9c1-602a-4a6a-89e1-e90b8f77d980","Type":"ContainerStarted","Data":"1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6"}
Nov 02 14:56:17.153946 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:17.153905    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hfx8" event={"ID":"c8287a73-bfc6-492d-8be0-109ba812e3af","Type":"ContainerStarted","Data":"b074541fd9ba8e3764ae411d0fb267d26880039230bbf8e6a71efb26fc32da78"}
Nov 02 14:56:17.198030 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:17.198000    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:56:17.198293 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:17.198245    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7585cfc69f-nwnpt" podStartSLOduration=1.5856405869999999 podStartE2EDuration="5.198229495s" totalImagesPullingTime="3.612588908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:17.19782478 +0000 UTC m=+29.455894253" watchObservedRunningTime="2026-11-02 14:56:17.198229495 +0000 UTC m=+29.456298948"
Nov 02 14:56:18.161306 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:18.161225    5041 generic.go:356] "Generic (PLEG): container finished" podID="c8287a73-bfc6-492d-8be0-109ba812e3af" containerID="25a2dd2d7207c2b90501489fd3cc79f1508db3cd44d23b0682746079bf570963" exitCode=0
Nov 02 14:56:18.161734 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:18.161309    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hfx8" event={"ID":"c8287a73-bfc6-492d-8be0-109ba812e3af","Type":"ContainerDied","Data":"25a2dd2d7207c2b90501489fd3cc79f1508db3cd44d23b0682746079bf570963"}
Nov 02 14:56:18.251951 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:18.251930    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gnfc9"
Nov 02 14:56:19.169958 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:19.169919    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hfx8" event={"ID":"c8287a73-bfc6-492d-8be0-109ba812e3af","Type":"ContainerStarted","Data":"e67e51805e8a91728f98ca1c2993f2f885ff744769961026751ca6d7cc86e1c7"}
Nov 02 14:56:19.170461 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:19.169984    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hfx8" event={"ID":"c8287a73-bfc6-492d-8be0-109ba812e3af","Type":"ContainerStarted","Data":"d2a2b6f81b7938be28c8b708286d91ed4fe248bed6481f59dbd6ba73edc2949b"}
Nov 02 14:56:19.217150 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:19.217017    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5hfx8" podStartSLOduration=2.65642813 podStartE2EDuration="4.216999031s" totalImagesPullingTime="1.477204906s" totalInitContainerRuntime="83.365995ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:19.215207879 +0000 UTC m=+31.473277343" watchObservedRunningTime="2026-11-02 14:56:19.216999031 +0000 UTC m=+31.475068484"
Nov 02 14:56:22.532195 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.532156    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c9fb8c55d-nrpph"]
Nov 02 14:56:22.535510 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.535484    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.586443 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.586412    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 14:56:22.588569 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.588528    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9fb8c55d-nrpph"]
Nov 02 14:56:22.637112 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637076    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzxz\" (UniqueName: \"kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637263 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637137    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637263 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637192    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637356 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637298    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637356 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637326    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637472 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637351    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637527 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637473    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.637527 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.637507    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.692591 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.692563    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:22.692756 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.692624    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:22.698731 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.698707    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:22.738631 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738587    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738795 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738693    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzxz\" (UniqueName: \"kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738795 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738736    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738795 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738762    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738940 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738796    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738940 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738822    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738940 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738845    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.738940 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.738899    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.739688 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.739664    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.739836 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.739738    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.739991 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.739966    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.740158 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.740133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.740468 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.740447    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.742459 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.742436    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.753951 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.753927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.767256 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.767237    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzxz\" (UniqueName: \"kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz\") pod \"console-5c9fb8c55d-nrpph\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") " pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:22.849300 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:22.849226    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:23.161303 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:23.161204    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9fb8c55d-nrpph"]
Nov 02 14:56:23.166241 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:23.166185    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16658d61_928c_4805_9548_ac52d4452028.slice/crio-612ad73238b29de1f03cc0edea47a73b6b3c4fd6aba17550d48fbe88564a920f WatchSource:0}: Error finding container 612ad73238b29de1f03cc0edea47a73b6b3c4fd6aba17550d48fbe88564a920f: Status 404 returned error can't find the container with id 612ad73238b29de1f03cc0edea47a73b6b3c4fd6aba17550d48fbe88564a920f
Nov 02 14:56:23.192187 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:23.192143    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fb8c55d-nrpph" event={"ID":"16658d61-928c-4805-9548-ac52d4452028","Type":"ContainerStarted","Data":"612ad73238b29de1f03cc0edea47a73b6b3c4fd6aba17550d48fbe88564a920f"}
Nov 02 14:56:23.197125 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:23.197101    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:56:24.198227 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:24.198188    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fb8c55d-nrpph" event={"ID":"16658d61-928c-4805-9548-ac52d4452028","Type":"ContainerStarted","Data":"df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed"}
Nov 02 14:56:25.207119 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:25.207068    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-r2nzt" event={"ID":"41a77928-03ff-4526-9e95-10c7ae70657f","Type":"ContainerStarted","Data":"1279a4de6ebcc3f5ad61d5df7f060a467357ae547ed5c87cfaff6d09c455ccbe"}
Nov 02 14:56:25.207669 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:25.207186    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:25.222229 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:25.222199    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-r2nzt"
Nov 02 14:56:25.271191 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:25.271142    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-r2nzt" podStartSLOduration=1.410045864 podStartE2EDuration="18.271125082s" totalImagesPullingTime="16.861079218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:25.259571929 +0000 UTC m=+37.517641380" watchObservedRunningTime="2026-11-02 14:56:25.271125082 +0000 UTC m=+37.529194533"
Nov 02 14:56:25.271389 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:25.271334    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c9fb8c55d-nrpph" podStartSLOduration=3.27132201 podStartE2EDuration="3.27132201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:24.25421834 +0000 UTC m=+36.512287789" watchObservedRunningTime="2026-11-02 14:56:25.27132201 +0000 UTC m=+37.529391463"
Nov 02 14:56:29.106137 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:29.106051    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b46b95748-pwxs9"
Nov 02 14:56:29.190819 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:29.190779    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f8679cb6-t6942"]
Nov 02 14:56:31.793430 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:31.793385    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c9fb8c55d-nrpph"]
Nov 02 14:56:31.851791 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:31.851754    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-849f477b5c-hbkwm"]
Nov 02 14:56:31.895521 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:31.895493    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:31.902652 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:31.902620    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-849f477b5c-hbkwm"]
Nov 02 14:56:32.055187 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055110    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055187 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055153    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhsl2\" (UniqueName: \"kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055428 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055347    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055428 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055408    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055539 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055438    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055539 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055497    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055541    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.055636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.055571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156008 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.155972    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156188 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156013    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhsl2\" (UniqueName: \"kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156188 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156188 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156108    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156188 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156169    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156506 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156473    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156708 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156684    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156858 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156835    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156922 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156878    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.156975 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.156937    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.160412 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.157707    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.160412 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.158324    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.160412 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.159358    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.166256 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.166230    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.168098 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.168076    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.180220 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.180194    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhsl2\" (UniqueName: \"kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2\") pod \"console-849f477b5c-hbkwm\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") " pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.212858 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.212798    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5" podUID="58e33558-be4f-4b8b-812e-f73322516ac0" containerName="registry" containerID="cri-o://c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d" gracePeriod=30
Nov 02 14:56:32.239139 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.239111    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:32.536671 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.536645    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:56:32.562464 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:32.562429    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a0b4f7_02d5_45f4_88e5_aad194263c32.slice/crio-6c7730d1892d2b6b85230e299a933ae27ef81de010849659d610e3e85361e934 WatchSource:0}: Error finding container 6c7730d1892d2b6b85230e299a933ae27ef81de010849659d610e3e85361e934: Status 404 returned error can't find the container with id 6c7730d1892d2b6b85230e299a933ae27ef81de010849659d610e3e85361e934
Nov 02 14:56:32.565976 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.565955    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-849f477b5c-hbkwm"]
Nov 02 14:56:32.660437 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660412    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660560 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660472    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660560 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660530    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660672 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660596    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660672 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660664    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660775 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660708    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660828 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.660828 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.660805    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") pod \"58e33558-be4f-4b8b-812e-f73322516ac0\" (UID: \"58e33558-be4f-4b8b-812e-f73322516ac0\") "
Nov 02 14:56:32.662261 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.661140    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:32.662261 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.661410    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:32.663119 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.663095    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:32.663345 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.663311    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "kube-api-access-rzf95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:32.663498 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.663481    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:32.663701 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.663679    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:32.664011 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.663759    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:32.668300 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.668274    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted" pod "58e33558-be4f-4b8b-812e-f73322516ac0" (UID: "58e33558-be4f-4b8b-812e-f73322516ac0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:32.762451 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762419    5041 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-registry-certificates\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762451 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762448    5041 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e33558-be4f-4b8b-812e-f73322516ac0-ca-trust-extracted\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762451 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762459    5041 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-installation-pull-secrets\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762697 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762469    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e33558-be4f-4b8b-812e-f73322516ac0-trusted-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762697 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762478    5041 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-bound-sa-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762697 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762486    5041 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e33558-be4f-4b8b-812e-f73322516ac0-image-registry-private-configuration\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762697 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762494    5041 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-registry-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.762697 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.762502    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzf95\" (UniqueName: \"kubernetes.io/projected/58e33558-be4f-4b8b-812e-f73322516ac0-kube-api-access-rzf95\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:32.850023 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:32.849946    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:33.274428 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.274391    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849f477b5c-hbkwm" event={"ID":"57a0b4f7-02d5-45f4-88e5-aad194263c32","Type":"ContainerStarted","Data":"724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d"}
Nov 02 14:56:33.274612 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.274442    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849f477b5c-hbkwm" event={"ID":"57a0b4f7-02d5-45f4-88e5-aad194263c32","Type":"ContainerStarted","Data":"6c7730d1892d2b6b85230e299a933ae27ef81de010849659d610e3e85361e934"}
Nov 02 14:56:33.275611 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.275583    5041 generic.go:356] "Generic (PLEG): container finished" podID="58e33558-be4f-4b8b-812e-f73322516ac0" containerID="c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d" exitCode=0
Nov 02 14:56:33.275719 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.275625    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5"
Nov 02 14:56:33.275719 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.275664    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5" event={"ID":"58e33558-be4f-4b8b-812e-f73322516ac0","Type":"ContainerDied","Data":"c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d"}
Nov 02 14:56:33.275719 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.275709    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588df7d7d7-sv5v5" event={"ID":"58e33558-be4f-4b8b-812e-f73322516ac0","Type":"ContainerDied","Data":"eeea2430725f0f107ecdd3c961d2f06ccdd6afc3612be52b67ad0ea0a22b283d"}
Nov 02 14:56:33.275873 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.275727    5041 scope.go:122] "RemoveContainer" containerID="c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d"
Nov 02 14:56:33.284347 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.284330    5041 scope.go:122] "RemoveContainer" containerID="c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d"
Nov 02 14:56:33.284615 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:56:33.284597    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d\": container with ID starting with c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d not found: ID does not exist" containerID="c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d"
Nov 02 14:56:33.284672 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.284623    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d"} err="failed to get container status \"c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d\": rpc error: code = NotFound desc = could not find container \"c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d\": container with ID starting with c9a2abd9ef1dad32612ae08679b5fc7e78ea852b65ad0c88d18d1ae18393d96d not found: ID does not exist"
Nov 02 14:56:33.317563 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.317525    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-849f477b5c-hbkwm" podStartSLOduration=2.317513449 podStartE2EDuration="2.317513449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:33.317189175 +0000 UTC m=+45.575258624" watchObservedRunningTime="2026-11-02 14:56:33.317513449 +0000 UTC m=+45.575582899"
Nov 02 14:56:33.353353 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.353327    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-588df7d7d7-sv5v5"]
Nov 02 14:56:33.363766 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.363740    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-588df7d7d7-sv5v5"]
Nov 02 14:56:33.885017 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:33.884985    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e33558-be4f-4b8b-812e-f73322516ac0" path="/var/lib/kubelet/pods/58e33558-be4f-4b8b-812e-f73322516ac0/volumes"
Nov 02 14:56:36.767331 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:56:36.767296    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:56:39.435220 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.435187    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-28f2f-m4r8k"]
Nov 02 14:56:39.435745 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.435433    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-28f2f-m4r8k" podUID="e1bc2365-fcbd-4dd1-9fda-bcad3e390603" containerName="insights-gathering" containerID="cri-o://95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883" gracePeriod=30
Nov 02 14:56:39.439879 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.439851    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-j6hws-tdl49"]
Nov 02 14:56:39.440105 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.440076    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-j6hws-tdl49" podUID="48c5d512-8f1d-4fbb-86d1-0651b52b965f" containerName="insights-gathering" containerID="cri-o://b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415" gracePeriod=30
Nov 02 14:56:39.765162 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.765131    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:56:39.779225 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.779203    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:39.930470 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930438    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") pod \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") "
Nov 02 14:56:39.930470 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930476    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path\") pod \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") "
Nov 02 14:56:39.930659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930541    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") pod \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") "
Nov 02 14:56:39.930659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930573    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") pod \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") "
Nov 02 14:56:39.930659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930639    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr\" (UniqueName: \"kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr\") pod \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") "
Nov 02 14:56:39.930805 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle\") pod \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") "
Nov 02 14:56:39.930805 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930771    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") pod \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\" (UID: \"48c5d512-8f1d-4fbb-86d1-0651b52b965f\") "
Nov 02 14:56:39.930914 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930839    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle\") pod \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\" (UID: \"e1bc2365-fcbd-4dd1-9fda-bcad3e390603\") "
Nov 02 14:56:39.930914 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930844    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path" pod "e1bc2365-fcbd-4dd1-9fda-bcad3e390603" (UID: "e1bc2365-fcbd-4dd1-9fda-bcad3e390603"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:39.931026 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.930916    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path" pod "48c5d512-8f1d-4fbb-86d1-0651b52b965f" (UID: "48c5d512-8f1d-4fbb-86d1-0651b52b965f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:39.931092 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931021    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle" pod "e1bc2365-fcbd-4dd1-9fda-bcad3e390603" (UID: "e1bc2365-fcbd-4dd1-9fda-bcad3e390603"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:39.931092 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931055    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle" pod "48c5d512-8f1d-4fbb-86d1-0651b52b965f" (UID: "48c5d512-8f1d-4fbb-86d1-0651b52b965f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:39.931218 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931198    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle" pod "48c5d512-8f1d-4fbb-86d1-0651b52b965f" (UID: "48c5d512-8f1d-4fbb-86d1-0651b52b965f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:39.931310 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931297    5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-archives-path\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:39.931382 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931302    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle" pod "e1bc2365-fcbd-4dd1-9fda-bcad3e390603" (UID: "e1bc2365-fcbd-4dd1-9fda-bcad3e390603"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:39.931382 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931312    5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/48c5d512-8f1d-4fbb-86d1-0651b52b965f-archives-path\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:39.931382 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931341    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-trusted-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:39.931382 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931351    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-service-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:39.931382 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.931377    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c5d512-8f1d-4fbb-86d1-0651b52b965f-service-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:39.932715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.932691    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr" pod "e1bc2365-fcbd-4dd1-9fda-bcad3e390603" (UID: "e1bc2365-fcbd-4dd1-9fda-bcad3e390603"). InnerVolumeSpecName "kube-api-access-4jgvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:39.932715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:39.932707    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn" pod "48c5d512-8f1d-4fbb-86d1-0651b52b965f" (UID: "48c5d512-8f1d-4fbb-86d1-0651b52b965f"). InnerVolumeSpecName "kube-api-access-wgrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:40.032231 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.032153    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrpn\" (UniqueName: \"kubernetes.io/projected/48c5d512-8f1d-4fbb-86d1-0651b52b965f-kube-api-access-wgrpn\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:40.032231 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.032182    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jgvr\" (UniqueName: \"kubernetes.io/projected/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-kube-api-access-4jgvr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:40.032231 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.032193    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bc2365-fcbd-4dd1-9fda-bcad3e390603-trusted-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:40.309542 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.309451    5041 generic.go:356] "Generic (PLEG): container finished" podID="e1bc2365-fcbd-4dd1-9fda-bcad3e390603" containerID="95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883" exitCode=2
Nov 02 14:56:40.309542 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.309495    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-28f2f-m4r8k"
Nov 02 14:56:40.309763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.309487    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-28f2f-m4r8k" event={"ID":"e1bc2365-fcbd-4dd1-9fda-bcad3e390603","Type":"ContainerDied","Data":"95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883"}
Nov 02 14:56:40.309763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.309647    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-28f2f-m4r8k" event={"ID":"e1bc2365-fcbd-4dd1-9fda-bcad3e390603","Type":"ContainerDied","Data":"81526780d50549142a5489c75317e4a90d43ec7272f2b2a4441e9beee48d74dd"}
Nov 02 14:56:40.309763 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.309672    5041 scope.go:122] "RemoveContainer" containerID="95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883"
Nov 02 14:56:40.310901 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.310882    5041 generic.go:356] "Generic (PLEG): container finished" podID="48c5d512-8f1d-4fbb-86d1-0651b52b965f" containerID="b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415" exitCode=2
Nov 02 14:56:40.310989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.310918    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6hws-tdl49"
Nov 02 14:56:40.311050 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.310924    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6hws-tdl49" event={"ID":"48c5d512-8f1d-4fbb-86d1-0651b52b965f","Type":"ContainerDied","Data":"b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415"}
Nov 02 14:56:40.311050 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.311022    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6hws-tdl49" event={"ID":"48c5d512-8f1d-4fbb-86d1-0651b52b965f","Type":"ContainerDied","Data":"572bcbb3f960193266fe399399fb3f42a85f1550bb19d3e59177341958f4973a"}
Nov 02 14:56:40.319730 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.319710    5041 scope.go:122] "RemoveContainer" containerID="95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883"
Nov 02 14:56:40.319999 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:56:40.319980    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883\": container with ID starting with 95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883 not found: ID does not exist" containerID="95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883"
Nov 02 14:56:40.320053 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.320006    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883"} err="failed to get container status \"95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883\": rpc error: code = NotFound desc = could not find container \"95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883\": container with ID starting with 95dd06ab4a3d1f0cf3339804fa5957234fcc22dcdd1546901dddbf234d733883 not found: ID does not exist"
Nov 02 14:56:40.320053 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.320030    5041 scope.go:122] "RemoveContainer" containerID="b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415"
Nov 02 14:56:40.328810 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.328786    5041 scope.go:122] "RemoveContainer" containerID="b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415"
Nov 02 14:56:40.329042 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:56:40.329024    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415\": container with ID starting with b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415 not found: ID does not exist" containerID="b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415"
Nov 02 14:56:40.329099 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.329049    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415"} err="failed to get container status \"b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415\": rpc error: code = NotFound desc = could not find container \"b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415\": container with ID starting with b98142c84b830ef8e458da0762d6836e43c801ffbc6091b41bcf0d3b36045415 not found: ID does not exist"
Nov 02 14:56:40.361860 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.361833    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-28f2f-m4r8k"]
Nov 02 14:56:40.373348 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.373322    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-28f2f-m4r8k"]
Nov 02 14:56:40.406535 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.406504    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-j6hws-tdl49"]
Nov 02 14:56:40.414267 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:40.414239    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-j6hws-tdl49"]
Nov 02 14:56:41.884780 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:41.884749    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c5d512-8f1d-4fbb-86d1-0651b52b965f" path="/var/lib/kubelet/pods/48c5d512-8f1d-4fbb-86d1-0651b52b965f/volumes"
Nov 02 14:56:41.885160 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:41.885123    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1bc2365-fcbd-4dd1-9fda-bcad3e390603" path="/var/lib/kubelet/pods/e1bc2365-fcbd-4dd1-9fda-bcad3e390603/volumes"
Nov 02 14:56:42.240123 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:42.240085    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:42.240326 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:42.240164    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:42.245007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:42.244985    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:42.325309 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:42.325277    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:56:42.399854 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:42.399814    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7585cfc69f-nwnpt"]
Nov 02 14:56:54.216981 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.216932    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerName="registry" containerID="cri-o://6966800081a2b11f9d38389930628bbef2c922084d08146a06656b94e10be85f" gracePeriod=30
Nov 02 14:56:54.374054 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.374027    5041 generic.go:356] "Generic (PLEG): container finished" podID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerID="6966800081a2b11f9d38389930628bbef2c922084d08146a06656b94e10be85f" exitCode=0
Nov 02 14:56:54.374181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.374062    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" event={"ID":"0f276b8e-7e4f-4980-a8a3-9fed6ee92685","Type":"ContainerDied","Data":"6966800081a2b11f9d38389930628bbef2c922084d08146a06656b94e10be85f"}
Nov 02 14:56:54.491262 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.491208    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:56:54.552937 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.552914    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553056 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.552962    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553056 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.552989    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553132 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553054    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553132 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553096    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553218 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553132    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553218 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553170    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553218 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553199    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") pod \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\" (UID: \"0f276b8e-7e4f-4980-a8a3-9fed6ee92685\") "
Nov 02 14:56:54.553636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553589    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:54.553636 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.553604    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:54.555404 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.555349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:54.555519 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.555438    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "kube-api-access-g464c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:54.555785 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.555756    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:54.555879 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.555853    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:54.556078 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.556059    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:54.560449 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.560263    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted" pod "0f276b8e-7e4f-4980-a8a3-9fed6ee92685" (UID: "0f276b8e-7e4f-4980-a8a3-9fed6ee92685"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:54.654202 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654181    5041 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-ca-trust-extracted\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654202 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654201    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-trusted-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654327 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654212    5041 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-certificates\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654327 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654221    5041 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-registry-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654327 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654231    5041 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-image-registry-private-configuration\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654327 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654241    5041 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-installation-pull-secrets\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654327 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654250    5041 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-bound-sa-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:54.654327 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:54.654258    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g464c\" (UniqueName: \"kubernetes.io/projected/0f276b8e-7e4f-4980-a8a3-9fed6ee92685-kube-api-access-g464c\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:55.384041 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:55.383995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f8679cb6-t6942" event={"ID":"0f276b8e-7e4f-4980-a8a3-9fed6ee92685","Type":"ContainerDied","Data":"07a6c27ea1147a00d429f97b293087ed2219c54143870b63c15e986899fea80c"}
Nov 02 14:56:55.384041 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:55.384026    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f8679cb6-t6942"
Nov 02 14:56:55.384508 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:55.384093    5041 scope.go:122] "RemoveContainer" containerID="6966800081a2b11f9d38389930628bbef2c922084d08146a06656b94e10be85f"
Nov 02 14:56:55.445378 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:55.445343    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f8679cb6-t6942"]
Nov 02 14:56:55.456535 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:55.456513    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f8679cb6-t6942"]
Nov 02 14:56:55.885294 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:55.885267    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" path="/var/lib/kubelet/pods/0f276b8e-7e4f-4980-a8a3-9fed6ee92685/volumes"
Nov 02 14:56:56.816346 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:56.816287    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c9fb8c55d-nrpph" podUID="16658d61-928c-4805-9548-ac52d4452028" containerName="console" containerID="cri-o://df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed" gracePeriod=15
Nov 02 14:56:57.093848 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.093794    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:57.174134 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174103    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174296 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174154    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174296 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174191    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174296 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174218    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174296 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174241    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174530 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174322    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174530 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz\" (UniqueName: \"kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174530 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174434    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp\") pod \"16658d61-928c-4805-9548-ac52d4452028\" (UID: \"16658d61-928c-4805-9548-ac52d4452028\") "
Nov 02 14:56:57.174683 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174561    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.174683 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174535    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.174753 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.174789 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174756    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.174789 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174775    5041 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-console-config\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.174889 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.174818    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:57.176249 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.175009    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.176726 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.176701    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.176876 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.176747    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "kube-api-access-trzxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:57.176876 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.176784    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert" pod "16658d61-928c-4805-9548-ac52d4452028" (UID: "16658d61-928c-4805-9548-ac52d4452028"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.275846 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.275820    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-trusted-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.275846 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.275845    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trzxz\" (UniqueName: \"kubernetes.io/projected/16658d61-928c-4805-9548-ac52d4452028-kube-api-access-trzxz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.275984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.275857    5041 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16658d61-928c-4805-9548-ac52d4452028-tmp\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.275984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.275866    5041 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-oauth-config\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.275984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.275876    5041 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16658d61-928c-4805-9548-ac52d4452028-oauth-serving-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.275984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.275884    5041 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16658d61-928c-4805-9548-ac52d4452028-console-serving-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.292777 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.292753    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b8d6dc47b-5js62"]
Nov 02 14:56:57.293069 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293057    5041 cpu_manager.go:414] "containerMap: removing container" podUID="58e33558-be4f-4b8b-812e-f73322516ac0" containerName="registry"
Nov 02 14:56:57.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293070    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e33558-be4f-4b8b-812e-f73322516ac0" containerName="registry"
Nov 02 14:56:57.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293080    5041 cpu_manager.go:414] "containerMap: removing container" podUID="16658d61-928c-4805-9548-ac52d4452028" containerName="console"
Nov 02 14:56:57.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293085    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16658d61-928c-4805-9548-ac52d4452028" containerName="console"
Nov 02 14:56:57.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293099    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerName="registry"
Nov 02 14:56:57.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293104    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerName="registry"
Nov 02 14:56:57.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293116    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e1bc2365-fcbd-4dd1-9fda-bcad3e390603" containerName="insights-gathering"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293122    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bc2365-fcbd-4dd1-9fda-bcad3e390603" containerName="insights-gathering"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293136    5041 cpu_manager.go:414] "containerMap: removing container" podUID="48c5d512-8f1d-4fbb-86d1-0651b52b965f" containerName="insights-gathering"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293140    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c5d512-8f1d-4fbb-86d1-0651b52b965f" containerName="insights-gathering"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293207    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c5d512-8f1d-4fbb-86d1-0651b52b965f" containerName="insights-gathering"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293217    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e33558-be4f-4b8b-812e-f73322516ac0" containerName="registry"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293223    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="16658d61-928c-4805-9548-ac52d4452028" containerName="console"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293231    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f276b8e-7e4f-4980-a8a3-9fed6ee92685" containerName="registry"
Nov 02 14:56:57.293283 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.293238    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1bc2365-fcbd-4dd1-9fda-bcad3e390603" containerName="insights-gathering"
Nov 02 14:56:57.322752 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.322726    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.324458 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.324398    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8d6dc47b-5js62"]
Nov 02 14:56:57.376566 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376499    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-trusted-ca-bundle\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376566 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376553    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21e33573-132d-4202-a84f-043149a00678-console-serving-cert\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376740 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376678    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e33573-132d-4202-a84f-043149a00678-tmp\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376740 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376722    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-console-config\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376823 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376759    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-oauth-serving-cert\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376857 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376835    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21e33573-132d-4202-a84f-043149a00678-console-oauth-config\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376890 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376862    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzlq\" (UniqueName: \"kubernetes.io/projected/21e33573-132d-4202-a84f-043149a00678-kube-api-access-8nzlq\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.376946 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.376903    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-service-ca\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.394480 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.394456    5041 generic.go:356] "Generic (PLEG): container finished" podID="16658d61-928c-4805-9548-ac52d4452028" containerID="df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed" exitCode=0
Nov 02 14:56:57.394574 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.394510    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9fb8c55d-nrpph"
Nov 02 14:56:57.394574 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.394521    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fb8c55d-nrpph" event={"ID":"16658d61-928c-4805-9548-ac52d4452028","Type":"ContainerDied","Data":"df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed"}
Nov 02 14:56:57.394574 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.394559    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9fb8c55d-nrpph" event={"ID":"16658d61-928c-4805-9548-ac52d4452028","Type":"ContainerDied","Data":"612ad73238b29de1f03cc0edea47a73b6b3c4fd6aba17550d48fbe88564a920f"}
Nov 02 14:56:57.394788 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.394578    5041 scope.go:122] "RemoveContainer" containerID="df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed"
Nov 02 14:56:57.408119 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.408098    5041 scope.go:122] "RemoveContainer" containerID="df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed"
Nov 02 14:56:57.408359 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:56:57.408342    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed\": container with ID starting with df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed not found: ID does not exist" containerID="df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed"
Nov 02 14:56:57.408435 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.408386    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed"} err="failed to get container status \"df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed\": rpc error: code = NotFound desc = could not find container \"df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed\": container with ID starting with df0366595beb68e8fca115a279fa00ac53f831eaf3d95b0a3669639b995630ed not found: ID does not exist"
Nov 02 14:56:57.446994 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.446971    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c9fb8c55d-nrpph"]
Nov 02 14:56:57.453547 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.453526    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c9fb8c55d-nrpph"]
Nov 02 14:56:57.477479 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477458    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21e33573-132d-4202-a84f-043149a00678-console-oauth-config\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477568 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477485    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzlq\" (UniqueName: \"kubernetes.io/projected/21e33573-132d-4202-a84f-043149a00678-kube-api-access-8nzlq\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477568 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477514    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-service-ca\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477568 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-trusted-ca-bundle\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477683 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477597    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21e33573-132d-4202-a84f-043149a00678-console-serving-cert\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477752 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477733    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e33573-132d-4202-a84f-043149a00678-tmp\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477810 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477793    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-console-config\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.477865 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.477832    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-oauth-serving-cert\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.478596 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.478134    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e33573-132d-4202-a84f-043149a00678-tmp\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.478596 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.478234    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-service-ca\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.478596 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.478489    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-oauth-serving-cert\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.478596 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.478539    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-trusted-ca-bundle\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.478596 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.478568    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21e33573-132d-4202-a84f-043149a00678-console-config\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.479866 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.479846    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21e33573-132d-4202-a84f-043149a00678-console-serving-cert\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.479934 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.479914    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21e33573-132d-4202-a84f-043149a00678-console-oauth-config\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.500633 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.500612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzlq\" (UniqueName: \"kubernetes.io/projected/21e33573-132d-4202-a84f-043149a00678-kube-api-access-8nzlq\") pod \"console-b8d6dc47b-5js62\" (UID: \"21e33573-132d-4202-a84f-043149a00678\") " pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.660029 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.659872    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:56:57.885011 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.884972    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16658d61-928c-4805-9548-ac52d4452028" path="/var/lib/kubelet/pods/16658d61-928c-4805-9548-ac52d4452028/volumes"
Nov 02 14:56:57.976513 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:57.976483    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8d6dc47b-5js62"]
Nov 02 14:56:58.024962 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:56:58.024932    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e33573_132d_4202_a84f_043149a00678.slice/crio-8d51b8aa0af19bc13787c369a0caa5941879c6377ee1543aca6dfda5983f092d WatchSource:0}: Error finding container 8d51b8aa0af19bc13787c369a0caa5941879c6377ee1543aca6dfda5983f092d: Status 404 returned error can't find the container with id 8d51b8aa0af19bc13787c369a0caa5941879c6377ee1543aca6dfda5983f092d
Nov 02 14:56:58.400460 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:58.400358    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8d6dc47b-5js62" event={"ID":"21e33573-132d-4202-a84f-043149a00678","Type":"ContainerStarted","Data":"f43d73880c0e82792a2211aef36007fde09cd7bd7c20a5fb530cbe92e28232f7"}
Nov 02 14:56:58.400606 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:58.400481    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8d6dc47b-5js62" event={"ID":"21e33573-132d-4202-a84f-043149a00678","Type":"ContainerStarted","Data":"8d51b8aa0af19bc13787c369a0caa5941879c6377ee1543aca6dfda5983f092d"}
Nov 02 14:56:58.453667 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:56:58.453621    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b8d6dc47b-5js62" podStartSLOduration=1.453609725 podStartE2EDuration="1.453609725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:58.442131526 +0000 UTC m=+70.700200977" watchObservedRunningTime="2026-11-02 14:56:58.453609725 +0000 UTC m=+70.711679175"
Nov 02 14:57:06.769186 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:57:06.769146    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:57:07.426413 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.426350    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7585cfc69f-nwnpt" podUID="7680b9c1-602a-4a6a-89e1-e90b8f77d980" containerName="console" containerID="cri-o://1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6" gracePeriod=15
Nov 02 14:57:07.660616 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.660583    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:57:07.660779 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.660643    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:57:07.665328 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.665304    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:57:07.703477 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.703426    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:57:07.759226 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759200    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759376 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759272    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759376 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759313    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759376 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759345    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759579 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759399    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759579 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759436    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759579 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759473    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z\" (UniqueName: \"kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z\") pod \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\" (UID: \"7680b9c1-602a-4a6a-89e1-e90b8f77d980\") "
Nov 02 14:57:07.759767 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.759740    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:07.760504 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.760002    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:07.760504 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.760101    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:07.760504 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.760217    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:07.760504 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.760237    5041 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-config\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:07.760504 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.760250    5041 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7680b9c1-602a-4a6a-89e1-e90b8f77d980-tmp\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:07.760823 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.760498    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:07.761824 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.761802    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:07.762035 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.762017    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:07.762714 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.762689    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z" pod "7680b9c1-602a-4a6a-89e1-e90b8f77d980" (UID: "7680b9c1-602a-4a6a-89e1-e90b8f77d980"). InnerVolumeSpecName "kube-api-access-v4w9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:07.860839 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.860817    5041 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-oauth-config\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:07.860839 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.860839    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4w9z\" (UniqueName: \"kubernetes.io/projected/7680b9c1-602a-4a6a-89e1-e90b8f77d980-kube-api-access-v4w9z\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:07.861171 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.860849    5041 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7680b9c1-602a-4a6a-89e1-e90b8f77d980-oauth-serving-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:07.861171 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:07.860858    5041 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7680b9c1-602a-4a6a-89e1-e90b8f77d980-console-serving-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:08.442481 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.442447    5041 generic.go:356] "Generic (PLEG): container finished" podID="7680b9c1-602a-4a6a-89e1-e90b8f77d980" containerID="1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6" exitCode=0
Nov 02 14:57:08.442659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.442509    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7585cfc69f-nwnpt"
Nov 02 14:57:08.442659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.442535    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585cfc69f-nwnpt" event={"ID":"7680b9c1-602a-4a6a-89e1-e90b8f77d980","Type":"ContainerDied","Data":"1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6"}
Nov 02 14:57:08.442659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.442607    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7585cfc69f-nwnpt" event={"ID":"7680b9c1-602a-4a6a-89e1-e90b8f77d980","Type":"ContainerDied","Data":"6e6c43a8590c0219659bc2f0f3da39c4c296f95cd53bee80f9e410e54bc7ffe1"}
Nov 02 14:57:08.442659 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.442627    5041 scope.go:122] "RemoveContainer" containerID="1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6"
Nov 02 14:57:08.446716 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.446697    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b8d6dc47b-5js62"
Nov 02 14:57:08.451087 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.451071    5041 scope.go:122] "RemoveContainer" containerID="1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6"
Nov 02 14:57:08.451380 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:57:08.451340    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6\": container with ID starting with 1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6 not found: ID does not exist" containerID="1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6"
Nov 02 14:57:08.451439 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.451384    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6"} err="failed to get container status \"1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6\": rpc error: code = NotFound desc = could not find container \"1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6\": container with ID starting with 1b02c2465872fc98212b864394bc749d2151e63c453b71f85fa99db642de42c6 not found: ID does not exist"
Nov 02 14:57:08.492469 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.492445    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7585cfc69f-nwnpt"]
Nov 02 14:57:08.503803 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.503785    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7585cfc69f-nwnpt"]
Nov 02 14:57:08.569702 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:08.569680    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-849f477b5c-hbkwm"]
Nov 02 14:57:09.884886 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:09.884851    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7680b9c1-602a-4a6a-89e1-e90b8f77d980" path="/var/lib/kubelet/pods/7680b9c1-602a-4a6a-89e1-e90b8f77d980/volumes"
Nov 02 14:57:33.590242 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.590201    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-849f477b5c-hbkwm" podUID="57a0b4f7-02d5-45f4-88e5-aad194263c32" containerName="console" containerID="cri-o://724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d" gracePeriod=15
Nov 02 14:57:33.833667 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.833644    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:57:33.943323 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943297    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2\" (UniqueName: \"kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943458 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943342    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943458 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943429    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943534 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943481    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943534 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943515    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943623 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943542    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943623 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943572    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp\" (UniqueName: \"kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943623 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943599    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca\") pod \"57a0b4f7-02d5-45f4-88e5-aad194263c32\" (UID: \"57a0b4f7-02d5-45f4-88e5-aad194263c32\") "
Nov 02 14:57:33.943914 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943886    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:33.944007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943928    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:33.944007 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.943980    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:33.944171 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.944147    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:33.944221 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.944153    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:33.945497 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.945477    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "kube-api-access-fhsl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:33.945651 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.945631    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:33.945779 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:33.945762    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config" pod "57a0b4f7-02d5-45f4-88e5-aad194263c32" (UID: "57a0b4f7-02d5-45f4-88e5-aad194263c32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044493    5041 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-config\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044516    5041 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-serving-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044530    5041 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-oauth-serving-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044545    5041 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57a0b4f7-02d5-45f4-88e5-aad194263c32-console-oauth-config\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044559    5041 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57a0b4f7-02d5-45f4-88e5-aad194263c32-tmp\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044572    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044586    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhsl2\" (UniqueName: \"kubernetes.io/projected/57a0b4f7-02d5-45f4-88e5-aad194263c32-kube-api-access-fhsl2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.044629 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.044599    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a0b4f7-02d5-45f4-88e5-aad194263c32-trusted-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 14:57:34.550559 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.550522    5041 generic.go:356] "Generic (PLEG): container finished" podID="57a0b4f7-02d5-45f4-88e5-aad194263c32" containerID="724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d" exitCode=0
Nov 02 14:57:34.550704 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.550588    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849f477b5c-hbkwm"
Nov 02 14:57:34.550704 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.550598    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849f477b5c-hbkwm" event={"ID":"57a0b4f7-02d5-45f4-88e5-aad194263c32","Type":"ContainerDied","Data":"724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d"}
Nov 02 14:57:34.550704 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.550671    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849f477b5c-hbkwm" event={"ID":"57a0b4f7-02d5-45f4-88e5-aad194263c32","Type":"ContainerDied","Data":"6c7730d1892d2b6b85230e299a933ae27ef81de010849659d610e3e85361e934"}
Nov 02 14:57:34.550704 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.550692    5041 scope.go:122] "RemoveContainer" containerID="724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d"
Nov 02 14:57:34.559308 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.559291    5041 scope.go:122] "RemoveContainer" containerID="724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d"
Nov 02 14:57:34.559568 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:57:34.559547    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d\": container with ID starting with 724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d not found: ID does not exist" containerID="724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d"
Nov 02 14:57:34.559621 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.559575    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d"} err="failed to get container status \"724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d\": rpc error: code = NotFound desc = could not find container \"724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d\": container with ID starting with 724445c67a01a35ee50797893e6c8529ec0b36551c430092cdce9c438ae4848d not found: ID does not exist"
Nov 02 14:57:34.600598 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.600572    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-849f477b5c-hbkwm"]
Nov 02 14:57:34.611812 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:34.611791    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-849f477b5c-hbkwm"]
Nov 02 14:57:35.885187 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:57:35.885153    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a0b4f7-02d5-45f4-88e5-aad194263c32" path="/var/lib/kubelet/pods/57a0b4f7-02d5-45f4-88e5-aad194263c32/volumes"
Nov 02 14:57:36.767711 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:57:36.767679    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:58:06.766565 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:58:06.766486    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:58:36.770324 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:58:36.770283    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:59:06.767412 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:59:06.767375    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:59:36.767171 ip-10-0-133-143 kubenswrapper[5041]: E1102 14:59:36.767091    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 14:59:46.217728 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.217692    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"]
Nov 02 14:59:46.218181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.218034    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7680b9c1-602a-4a6a-89e1-e90b8f77d980" containerName="console"
Nov 02 14:59:46.218181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.218045    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7680b9c1-602a-4a6a-89e1-e90b8f77d980" containerName="console"
Nov 02 14:59:46.218181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.218058    5041 cpu_manager.go:414] "containerMap: removing container" podUID="57a0b4f7-02d5-45f4-88e5-aad194263c32" containerName="console"
Nov 02 14:59:46.218181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.218064    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a0b4f7-02d5-45f4-88e5-aad194263c32" containerName="console"
Nov 02 14:59:46.218181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.218143    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7680b9c1-602a-4a6a-89e1-e90b8f77d980" containerName="console"
Nov 02 14:59:46.218181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.218151    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a0b4f7-02d5-45f4-88e5-aad194263c32" containerName="console"
Nov 02 14:59:46.220230 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.220210    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.242320 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.242297    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-x2pbj\""
Nov 02 14:59:46.242429 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.242417    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 14:59:46.242532 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.242512    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 14:59:46.243779 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.243762    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"]
Nov 02 14:59:46.390403 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.390351    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzzk\" (UniqueName: \"kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.390564 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.390466    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.390564 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.390518    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.491324 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.491252    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.491324 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.491293    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.491502 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.491335    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzzk\" (UniqueName: \"kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.491641 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.491624    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.491692 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.491651    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.515607 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.515578    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzzk\" (UniqueName: \"kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.533402 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.533381    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 14:59:46.829236 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.829168    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"]
Nov 02 14:59:46.833082 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:46.833057    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03a1754_2282_4091_b7dd_c7ee97f6153a.slice/crio-267bec7f7adb0e29885ad214cbf6e2daf32641e37fb45c8f8dff26231e77f114 WatchSource:0}: Error finding container 267bec7f7adb0e29885ad214cbf6e2daf32641e37fb45c8f8dff26231e77f114: Status 404 returned error can't find the container with id 267bec7f7adb0e29885ad214cbf6e2daf32641e37fb45c8f8dff26231e77f114
Nov 02 14:59:46.907074 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.907051    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"]
Nov 02 14:59:46.910282 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.910260    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:46.922717 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.922698    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tmg9h\""
Nov 02 14:59:46.933929 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:46.933905    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"]
Nov 02 14:59:47.079414 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.079318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td" event={"ID":"f03a1754-2282-4091-b7dd-c7ee97f6153a","Type":"ContainerStarted","Data":"267bec7f7adb0e29885ad214cbf6e2daf32641e37fb45c8f8dff26231e77f114"}
Nov 02 14:59:47.095673 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.095642    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.095805 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.095698    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcs4\" (UniqueName: \"kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.095805 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.095791    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.196647 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.196615    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcs4\" (UniqueName: \"kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.196767 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.196674    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.196767 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.196712    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.197082 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.197062    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.197144 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.197117    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.218444 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.218419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcs4\" (UniqueName: \"kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.223041 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.223020    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 14:59:47.524028 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:47.523991    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"]
Nov 02 14:59:47.526798 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:47.526769    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e8b8ec_bd41_43a6_8e01_1b56e7f2d911.slice/crio-8c3f69bc1f1c1eaff787ea6d10ed05bfcad5d271f93cd916d518bbb441937227 WatchSource:0}: Error finding container 8c3f69bc1f1c1eaff787ea6d10ed05bfcad5d271f93cd916d518bbb441937227: Status 404 returned error can't find the container with id 8c3f69bc1f1c1eaff787ea6d10ed05bfcad5d271f93cd916d518bbb441937227
Nov 02 14:59:48.088984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:48.088944    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7" event={"ID":"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911","Type":"ContainerStarted","Data":"8c3f69bc1f1c1eaff787ea6d10ed05bfcad5d271f93cd916d518bbb441937227"}
Nov 02 14:59:51.065224 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.065189    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"]
Nov 02 14:59:51.068679 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.068649    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.090773 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.090750    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qjp47\""
Nov 02 14:59:51.092476 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.092453    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"]
Nov 02 14:59:51.132230 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.131982    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.132230 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.132039    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.132230 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.132097    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxbn\" (UniqueName: \"kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.233603 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.233558    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.233789 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.233610    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.233789 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.233657    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxbn\" (UniqueName: \"kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.234023 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.233988    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.234023 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.234010    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.256323 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.256294    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxbn\" (UniqueName: \"kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.385568 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.385485    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 14:59:51.689252 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:51.688280    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"]
Nov 02 14:59:51.692671 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:51.692641    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adcaa6c_1e09_4b5d_b285_e4041ae58644.slice/crio-5d6bc01988fe9c94c9003495b8fcb013a24ccc20db340992da74778ae19501b5 WatchSource:0}: Error finding container 5d6bc01988fe9c94c9003495b8fcb013a24ccc20db340992da74778ae19501b5: Status 404 returned error can't find the container with id 5d6bc01988fe9c94c9003495b8fcb013a24ccc20db340992da74778ae19501b5
Nov 02 14:59:52.111314 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:52.111227    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp" event={"ID":"7adcaa6c-1e09-4b5d-b285-e4041ae58644","Type":"ContainerStarted","Data":"5d6bc01988fe9c94c9003495b8fcb013a24ccc20db340992da74778ae19501b5"}
Nov 02 14:59:54.028432 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.028397    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"]
Nov 02 14:59:54.031599 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.031576    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.044181 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.044155    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8bq5t\""
Nov 02 14:59:54.055251 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.055223    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"]
Nov 02 14:59:54.066989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.064831    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"]
Nov 02 14:59:54.066989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.065696    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.066989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.065763    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nndb\" (UniqueName: \"kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.066989 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.065876    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.068162 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.068143    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.101284 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.101254    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"]
Nov 02 14:59:54.105098 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.105075    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"]
Nov 02 14:59:54.109241 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.109211    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.124311 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.124287    5041 generic.go:356] "Generic (PLEG): container finished" podID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerID="d7fd765caee0cd6e227a99ad8024e781d1a0b42b6e18e39d00201d9146ca709d" exitCode=0
Nov 02 14:59:54.124486 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.124463    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td" event={"ID":"f03a1754-2282-4091-b7dd-c7ee97f6153a","Type":"ContainerDied","Data":"d7fd765caee0cd6e227a99ad8024e781d1a0b42b6e18e39d00201d9146ca709d"}
Nov 02 14:59:54.130434 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.130406    5041 generic.go:356] "Generic (PLEG): container finished" podID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerID="f4888a18d715cb0a8214d176b5a5b33b2713e6d3a78b414f08ef37848ddffba7" exitCode=0
Nov 02 14:59:54.130537 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.130462    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7" event={"ID":"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911","Type":"ContainerDied","Data":"f4888a18d715cb0a8214d176b5a5b33b2713e6d3a78b414f08ef37848ddffba7"}
Nov 02 14:59:54.131332 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.131309    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"]
Nov 02 14:59:54.143854 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.143834    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"]
Nov 02 14:59:54.144611 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.144591    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qvj8m\""
Nov 02 14:59:54.144701 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.144681    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7qqk4\""
Nov 02 14:59:54.146795 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.146773    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.167179 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167154    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.167276 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167207    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.167331 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167271    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.167389 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167348    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.167441 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167423    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nndb\" (UniqueName: \"kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.167506 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167489    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.167544 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167494    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.167578 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167545    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.167578 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167572    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.167648 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167621    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fv2b\" (UniqueName: \"kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.167690 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167644    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566tq\" (UniqueName: \"kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.167690 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167683    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.167760 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167703    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwd8p\" (UniqueName: \"kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.167799 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.167761    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.169523 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.169501    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"]
Nov 02 14:59:54.189618 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.189562    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nndb\" (UniqueName: \"kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.268665 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268632    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.268833 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268690    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fv2b\" (UniqueName: \"kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.268833 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268768    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566tq\" (UniqueName: \"kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.268833 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268808    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.269019 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268838    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwd8p\" (UniqueName: \"kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.269019 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268864    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.269019 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268886    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.269019 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268916    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.269019 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.268978    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.269281 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.269031    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.269339 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.269281    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.269339 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.269289    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.269494 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.269478    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.269589 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.269499    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.269715 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.269696    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.290718 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.290693    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566tq\" (UniqueName: \"kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.291022 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.291000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fv2b\" (UniqueName: \"kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.291086 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.291003    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwd8p\" (UniqueName: \"kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.348103 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.348080    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 14:59:54.432755 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.432725    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 14:59:54.439941 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.439877    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 14:59:54.463041 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.462856    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 14:59:54.646451 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.646420    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"]
Nov 02 14:59:54.648986 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:54.648959    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241b0915_97ee_43ac_9f28_01d977ccc050.slice/crio-9fec01f7a22eb7a97b3868083324b0703a73f4e6a5a7751252cdeaca45017a2b WatchSource:0}: Error finding container 9fec01f7a22eb7a97b3868083324b0703a73f4e6a5a7751252cdeaca45017a2b: Status 404 returned error can't find the container with id 9fec01f7a22eb7a97b3868083324b0703a73f4e6a5a7751252cdeaca45017a2b
Nov 02 14:59:54.747909 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.746165    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"]
Nov 02 14:59:54.749055 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:54.749008    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ccfe58_be3f_47b1_a7da_b8781bd18257.slice/crio-54805bc58d2d115ef6f74be451bd7b68bf23b74b43b2f01232754c3d06ab91ad WatchSource:0}: Error finding container 54805bc58d2d115ef6f74be451bd7b68bf23b74b43b2f01232754c3d06ab91ad: Status 404 returned error can't find the container with id 54805bc58d2d115ef6f74be451bd7b68bf23b74b43b2f01232754c3d06ab91ad
Nov 02 14:59:54.763331 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.763087    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"]
Nov 02 14:59:54.768918 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:54.768892    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea277ef9_22d8_4415_8225_a7c14d5e477e.slice/crio-bff272e3b002c82648d7c163bcfd052c111c01064b8753dec4f12660bce0b2db WatchSource:0}: Error finding container bff272e3b002c82648d7c163bcfd052c111c01064b8753dec4f12660bce0b2db: Status 404 returned error can't find the container with id bff272e3b002c82648d7c163bcfd052c111c01064b8753dec4f12660bce0b2db
Nov 02 14:59:54.794322 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:54.794295    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"]
Nov 02 14:59:54.813566 ip-10-0-133-143 kubenswrapper[5041]: W1102 14:59:54.813545    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaebe4d4_9dc7_4e66_b35c_7b2ef4bd704e.slice/crio-bd1c80865903908cfa88a28c4a258598e8c6bfb1598b26de6bfcb7c3aca6e286 WatchSource:0}: Error finding container bd1c80865903908cfa88a28c4a258598e8c6bfb1598b26de6bfcb7c3aca6e286: Status 404 returned error can't find the container with id bd1c80865903908cfa88a28c4a258598e8c6bfb1598b26de6bfcb7c3aca6e286
Nov 02 14:59:55.138572 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.138538    5041 generic.go:356] "Generic (PLEG): container finished" podID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerID="ced9cbb16a2a9cb86cc9b4d4598fa50ffc704a8e1e78af78e4c292802f37cecc" exitCode=0
Nov 02 14:59:55.138984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.138631    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" event={"ID":"ea277ef9-22d8-4415-8225-a7c14d5e477e","Type":"ContainerDied","Data":"ced9cbb16a2a9cb86cc9b4d4598fa50ffc704a8e1e78af78e4c292802f37cecc"}
Nov 02 14:59:55.138984 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.138682    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" event={"ID":"ea277ef9-22d8-4415-8225-a7c14d5e477e","Type":"ContainerStarted","Data":"bff272e3b002c82648d7c163bcfd052c111c01064b8753dec4f12660bce0b2db"}
Nov 02 14:59:55.142091 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.141513    5041 generic.go:356] "Generic (PLEG): container finished" podID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerID="266109ae2088c28dcd29373eef2a97112c1e9f8fbc062d4f80a9708facb2a6c1" exitCode=0
Nov 02 14:59:55.142091 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.141761    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8" event={"ID":"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e","Type":"ContainerDied","Data":"266109ae2088c28dcd29373eef2a97112c1e9f8fbc062d4f80a9708facb2a6c1"}
Nov 02 14:59:55.142091 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.141810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8" event={"ID":"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e","Type":"ContainerStarted","Data":"bd1c80865903908cfa88a28c4a258598e8c6bfb1598b26de6bfcb7c3aca6e286"}
Nov 02 14:59:55.143633 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.143609    5041 generic.go:356] "Generic (PLEG): container finished" podID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerID="644ad17486ee2dba8c5217d34dd6e36c0d9c06e754906cb57ba0f2fc752cc7d3" exitCode=0
Nov 02 14:59:55.143724 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.143693    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" event={"ID":"50ccfe58-be3f-47b1-a7da-b8781bd18257","Type":"ContainerDied","Data":"644ad17486ee2dba8c5217d34dd6e36c0d9c06e754906cb57ba0f2fc752cc7d3"}
Nov 02 14:59:55.143783 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.143734    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" event={"ID":"50ccfe58-be3f-47b1-a7da-b8781bd18257","Type":"ContainerStarted","Data":"54805bc58d2d115ef6f74be451bd7b68bf23b74b43b2f01232754c3d06ab91ad"}
Nov 02 14:59:55.146956 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.146631    5041 generic.go:356] "Generic (PLEG): container finished" podID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerID="8e9915a12dca7721cae94edfd181c9cb3d6ef8f0005c14f3c3a63e9558e0909e" exitCode=0
Nov 02 14:59:55.146956 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.146706    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td" event={"ID":"f03a1754-2282-4091-b7dd-c7ee97f6153a","Type":"ContainerDied","Data":"8e9915a12dca7721cae94edfd181c9cb3d6ef8f0005c14f3c3a63e9558e0909e"}
Nov 02 14:59:55.149587 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.149414    5041 generic.go:356] "Generic (PLEG): container finished" podID="241b0915-97ee-43ac-9f28-01d977ccc050" containerID="09f955b30c369cc1569c2b503d96cbd29041cb4c4e07613b8532dd681956440e" exitCode=0
Nov 02 14:59:55.149587 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.149461    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt" event={"ID":"241b0915-97ee-43ac-9f28-01d977ccc050","Type":"ContainerDied","Data":"09f955b30c369cc1569c2b503d96cbd29041cb4c4e07613b8532dd681956440e"}
Nov 02 14:59:55.149587 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:55.149514    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt" event={"ID":"241b0915-97ee-43ac-9f28-01d977ccc050","Type":"ContainerStarted","Data":"9fec01f7a22eb7a97b3868083324b0703a73f4e6a5a7751252cdeaca45017a2b"}
Nov 02 14:59:56.159066 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.159036    5041 generic.go:356] "Generic (PLEG): container finished" podID="241b0915-97ee-43ac-9f28-01d977ccc050" containerID="3135747537ef863f7c8d18288e44bb5befe1f467ba529aa33dc71133132af78c" exitCode=0
Nov 02 14:59:56.159468 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.159166    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt" event={"ID":"241b0915-97ee-43ac-9f28-01d977ccc050","Type":"ContainerDied","Data":"3135747537ef863f7c8d18288e44bb5befe1f467ba529aa33dc71133132af78c"}
Nov 02 14:59:56.161654 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.161625    5041 generic.go:356] "Generic (PLEG): container finished" podID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerID="2afc83735ce3fb20b8385df894b5b886fdd08b974ba495e3121a3fd1f30ca026" exitCode=0
Nov 02 14:59:56.161764 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.161720    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" event={"ID":"ea277ef9-22d8-4415-8225-a7c14d5e477e","Type":"ContainerDied","Data":"2afc83735ce3fb20b8385df894b5b886fdd08b974ba495e3121a3fd1f30ca026"}
Nov 02 14:59:56.165712 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.164278    5041 generic.go:356] "Generic (PLEG): container finished" podID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerID="f86b8e89623fda05267b43210d14e30481bbdf5e9d59d21d72446e06df68f65c" exitCode=0
Nov 02 14:59:56.165712 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.164349    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8" event={"ID":"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e","Type":"ContainerDied","Data":"f86b8e89623fda05267b43210d14e30481bbdf5e9d59d21d72446e06df68f65c"}
Nov 02 14:59:56.167556 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.167422    5041 generic.go:356] "Generic (PLEG): container finished" podID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerID="f73bafce8dea87729c743e10b7427bb0cc0e7ccd1893e72ef755bfa0091247a0" exitCode=0
Nov 02 14:59:56.167556 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.167510    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" event={"ID":"50ccfe58-be3f-47b1-a7da-b8781bd18257","Type":"ContainerDied","Data":"f73bafce8dea87729c743e10b7427bb0cc0e7ccd1893e72ef755bfa0091247a0"}
Nov 02 14:59:56.169656 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.169623    5041 generic.go:356] "Generic (PLEG): container finished" podID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerID="8f0272bcc8087d9a28a415052a6be00b15d9d419a3f868c8160127fd431ac068" exitCode=0
Nov 02 14:59:56.169754 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:56.169701    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp" event={"ID":"7adcaa6c-1e09-4b5d-b285-e4041ae58644","Type":"ContainerDied","Data":"8f0272bcc8087d9a28a415052a6be00b15d9d419a3f868c8160127fd431ac068"}
Nov 02 14:59:57.181443 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:57.180241    5041 generic.go:356] "Generic (PLEG): container finished" podID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerID="5bf0e36b2e192fd38e8527519687baa611f5a050230e750df6264d1b3ff3572b" exitCode=0
Nov 02 14:59:57.182187 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:57.180358    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7" event={"ID":"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911","Type":"ContainerDied","Data":"5bf0e36b2e192fd38e8527519687baa611f5a050230e750df6264d1b3ff3572b"}
Nov 02 14:59:57.187841 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:57.187809    5041 generic.go:356] "Generic (PLEG): container finished" podID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerID="28209cc407da911b557fe4f36c300e7f166cfd95c169905dc7a074ed342d4540" exitCode=0
Nov 02 14:59:57.187930 ip-10-0-133-143 kubenswrapper[5041]: I1102 14:59:57.187914    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp" event={"ID":"7adcaa6c-1e09-4b5d-b285-e4041ae58644","Type":"ContainerDied","Data":"28209cc407da911b557fe4f36c300e7f166cfd95c169905dc7a074ed342d4540"}
Nov 02 15:00:06.798816 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:00:06.798730    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:00:09.305063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:09.305021    5041 generic.go:356] "Generic (PLEG): container finished" podID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerID="0fad4021457086fe6e033a1798d72f67ecbb25b5e966be5a21f9be964ae339ce" exitCode=0
Nov 02 15:00:09.305574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:09.305159    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td" event={"ID":"f03a1754-2282-4091-b7dd-c7ee97f6153a","Type":"ContainerDied","Data":"0fad4021457086fe6e033a1798d72f67ecbb25b5e966be5a21f9be964ae339ce"}
Nov 02 15:00:10.524181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.524146    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 15:00:10.633603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.633579    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk\" (UniqueName: \"kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk\") pod \"f03a1754-2282-4091-b7dd-c7ee97f6153a\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") "
Nov 02 15:00:10.633733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.633688    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle\") pod \"f03a1754-2282-4091-b7dd-c7ee97f6153a\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") "
Nov 02 15:00:10.633804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.633745    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util\") pod \"f03a1754-2282-4091-b7dd-c7ee97f6153a\" (UID: \"f03a1754-2282-4091-b7dd-c7ee97f6153a\") "
Nov 02 15:00:10.634191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.634147    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle" pod "f03a1754-2282-4091-b7dd-c7ee97f6153a" (UID: "f03a1754-2282-4091-b7dd-c7ee97f6153a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:10.636484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.636451    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk" pod "f03a1754-2282-4091-b7dd-c7ee97f6153a" (UID: "f03a1754-2282-4091-b7dd-c7ee97f6153a"). InnerVolumeSpecName "kube-api-access-vtzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:10.640335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.640309    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util" pod "f03a1754-2282-4091-b7dd-c7ee97f6153a" (UID: "f03a1754-2282-4091-b7dd-c7ee97f6153a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:10.735251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.735218    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:10.735251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.735252    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f03a1754-2282-4091-b7dd-c7ee97f6153a-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:10.735496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:10.735270    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzzk\" (UniqueName: \"kubernetes.io/projected/f03a1754-2282-4091-b7dd-c7ee97f6153a-kube-api-access-vtzzk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:11.325588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.325555    5041 generic.go:356] "Generic (PLEG): container finished" podID="241b0915-97ee-43ac-9f28-01d977ccc050" containerID="04bf45ce7e5f063964692ab1c05d0077a4e1e443eaa59fe0a86075c1a593bee0" exitCode=0
Nov 02 15:00:11.325687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.325637    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt" event={"ID":"241b0915-97ee-43ac-9f28-01d977ccc050","Type":"ContainerDied","Data":"04bf45ce7e5f063964692ab1c05d0077a4e1e443eaa59fe0a86075c1a593bee0"}
Nov 02 15:00:11.329357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.329332    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" event={"ID":"ea277ef9-22d8-4415-8225-a7c14d5e477e","Type":"ContainerStarted","Data":"6d9d0ab857c1515d1a8830b876cb70239a6f6b2c0c48d33241335a8a638864a7"}
Nov 02 15:00:11.332298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.332274    5041 generic.go:356] "Generic (PLEG): container finished" podID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerID="0b1d926c4d08b8f4944ac048e4fb662fa86da404c0cb70fc592e94a00e343426" exitCode=0
Nov 02 15:00:11.332397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.332345    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8" event={"ID":"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e","Type":"ContainerDied","Data":"0b1d926c4d08b8f4944ac048e4fb662fa86da404c0cb70fc592e94a00e343426"}
Nov 02 15:00:11.335451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.335386    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" event={"ID":"50ccfe58-be3f-47b1-a7da-b8781bd18257","Type":"ContainerStarted","Data":"d1b58137ae14be20ae4a9871563131275c98708e7cb42e6d7632ab527f87201f"}
Nov 02 15:00:11.337455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.337400    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td" event={"ID":"f03a1754-2282-4091-b7dd-c7ee97f6153a","Type":"ContainerDied","Data":"267bec7f7adb0e29885ad214cbf6e2daf32641e37fb45c8f8dff26231e77f114"}
Nov 02 15:00:11.337455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.337432    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="267bec7f7adb0e29885ad214cbf6e2daf32641e37fb45c8f8dff26231e77f114"
Nov 02 15:00:11.337544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.337495    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z29td"
Nov 02 15:00:11.402421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.402357    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" podStartSLOduration=1.499262373 podStartE2EDuration="17.402343113s" totalImagesPullingTime="15.90308074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:11.401023474 +0000 UTC m=+263.659092927" watchObservedRunningTime="2026-11-02 15:00:11.402343113 +0000 UTC m=+263.660412562"
Nov 02 15:00:11.485257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:11.484433    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" podStartSLOduration=1.8248276319999999 podStartE2EDuration="17.48441434s" totalImagesPullingTime="15.659586708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:11.441857069 +0000 UTC m=+263.699926521" watchObservedRunningTime="2026-11-02 15:00:11.48441434 +0000 UTC m=+263.742483884"
Nov 02 15:00:12.343203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.343130    5041 generic.go:356] "Generic (PLEG): container finished" podID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerID="c220356dde4e61c159323b60d8b2040ac4b9149ca93013db33e8ad8a093ca7d3" exitCode=0
Nov 02 15:00:12.343641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.343218    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7" event={"ID":"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911","Type":"ContainerDied","Data":"c220356dde4e61c159323b60d8b2040ac4b9149ca93013db33e8ad8a093ca7d3"}
Nov 02 15:00:12.345354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.345330    5041 generic.go:356] "Generic (PLEG): container finished" podID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerID="6d9d0ab857c1515d1a8830b876cb70239a6f6b2c0c48d33241335a8a638864a7" exitCode=0
Nov 02 15:00:12.345472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.345419    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" event={"ID":"ea277ef9-22d8-4415-8225-a7c14d5e477e","Type":"ContainerDied","Data":"6d9d0ab857c1515d1a8830b876cb70239a6f6b2c0c48d33241335a8a638864a7"}
Nov 02 15:00:12.347427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.347408    5041 generic.go:356] "Generic (PLEG): container finished" podID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerID="d1b58137ae14be20ae4a9871563131275c98708e7cb42e6d7632ab527f87201f" exitCode=0
Nov 02 15:00:12.347529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.347472    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" event={"ID":"50ccfe58-be3f-47b1-a7da-b8781bd18257","Type":"ContainerDied","Data":"d1b58137ae14be20ae4a9871563131275c98708e7cb42e6d7632ab527f87201f"}
Nov 02 15:00:12.349324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.349302    5041 generic.go:356] "Generic (PLEG): container finished" podID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerID="81a302dfbf29807c73d81cd97a055995ab60e4c6697d0a992361d4d59f0b76a9" exitCode=0
Nov 02 15:00:12.349427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.349387    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp" event={"ID":"7adcaa6c-1e09-4b5d-b285-e4041ae58644","Type":"ContainerDied","Data":"81a302dfbf29807c73d81cd97a055995ab60e4c6697d0a992361d4d59f0b76a9"}
Nov 02 15:00:12.501076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.501055    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 15:00:12.503782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.503761    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 15:00:12.649980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.649875    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle\") pod \"241b0915-97ee-43ac-9f28-01d977ccc050\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") "
Nov 02 15:00:12.649980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.649933    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util\") pod \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") "
Nov 02 15:00:12.649980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.649970    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p\" (UniqueName: \"kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p\") pod \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") "
Nov 02 15:00:12.650226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.650018    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb\" (UniqueName: \"kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb\") pod \"241b0915-97ee-43ac-9f28-01d977ccc050\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") "
Nov 02 15:00:12.650226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.650085    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util\") pod \"241b0915-97ee-43ac-9f28-01d977ccc050\" (UID: \"241b0915-97ee-43ac-9f28-01d977ccc050\") "
Nov 02 15:00:12.650226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.650167    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle\") pod \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\" (UID: \"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e\") "
Nov 02 15:00:12.650581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.650551    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle" pod "241b0915-97ee-43ac-9f28-01d977ccc050" (UID: "241b0915-97ee-43ac-9f28-01d977ccc050"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:12.650762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.650732    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle" pod "daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" (UID: "daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:12.652184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.652155    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb" pod "241b0915-97ee-43ac-9f28-01d977ccc050" (UID: "241b0915-97ee-43ac-9f28-01d977ccc050"). InnerVolumeSpecName "kube-api-access-4nndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:12.652524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.652495    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p" pod "daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" (UID: "daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e"). InnerVolumeSpecName "kube-api-access-rwd8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:12.655673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.655646    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util" pod "daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" (UID: "daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:12.656310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.656285    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util" pod "241b0915-97ee-43ac-9f28-01d977ccc050" (UID: "241b0915-97ee-43ac-9f28-01d977ccc050"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:12.751652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.751628    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nndb\" (UniqueName: \"kubernetes.io/projected/241b0915-97ee-43ac-9f28-01d977ccc050-kube-api-access-4nndb\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:12.751652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.751650    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:12.751774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.751662    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:12.751774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.751675    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/241b0915-97ee-43ac-9f28-01d977ccc050-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:12.751774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.751683    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:12.751774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:12.751691    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwd8p\" (UniqueName: \"kubernetes.io/projected/daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e-kube-api-access-rwd8p\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.356018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.355987    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8"
Nov 02 15:00:13.356496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.356014    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g54d8" event={"ID":"daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e","Type":"ContainerDied","Data":"bd1c80865903908cfa88a28c4a258598e8c6bfb1598b26de6bfcb7c3aca6e286"}
Nov 02 15:00:13.356496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.356074    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1c80865903908cfa88a28c4a258598e8c6bfb1598b26de6bfcb7c3aca6e286"
Nov 02 15:00:13.358138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.358114    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt" event={"ID":"241b0915-97ee-43ac-9f28-01d977ccc050","Type":"ContainerDied","Data":"9fec01f7a22eb7a97b3868083324b0703a73f4e6a5a7751252cdeaca45017a2b"}
Nov 02 15:00:13.358248 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.358145    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccxkgt"
Nov 02 15:00:13.358248 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.358154    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fec01f7a22eb7a97b3868083324b0703a73f4e6a5a7751252cdeaca45017a2b"
Nov 02 15:00:13.507283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.507260    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 15:00:13.554664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.554600    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 15:00:13.559755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.559734    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 15:00:13.562862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.562843    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 15:00:13.657081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657016    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle\") pod \"ea277ef9-22d8-4415-8225-a7c14d5e477e\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") "
Nov 02 15:00:13.657202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657086    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b\" (UniqueName: \"kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b\") pod \"ea277ef9-22d8-4415-8225-a7c14d5e477e\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") "
Nov 02 15:00:13.657202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657131    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle\") pod \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") "
Nov 02 15:00:13.657202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657162    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util\") pod \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") "
Nov 02 15:00:13.657202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657194    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn\" (UniqueName: \"kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn\") pod \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") "
Nov 02 15:00:13.657409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657226    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle\") pod \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\" (UID: \"7adcaa6c-1e09-4b5d-b285-e4041ae58644\") "
Nov 02 15:00:13.657409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657268    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle\") pod \"50ccfe58-be3f-47b1-a7da-b8781bd18257\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") "
Nov 02 15:00:13.657507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657470    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq\" (UniqueName: \"kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq\") pod \"50ccfe58-be3f-47b1-a7da-b8781bd18257\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") "
Nov 02 15:00:13.657565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657531    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util\") pod \"ea277ef9-22d8-4415-8225-a7c14d5e477e\" (UID: \"ea277ef9-22d8-4415-8225-a7c14d5e477e\") "
Nov 02 15:00:13.657617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657592    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util\") pod \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") "
Nov 02 15:00:13.657681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657629    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4\" (UniqueName: \"kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4\") pod \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\" (UID: \"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911\") "
Nov 02 15:00:13.657681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util\") pod \"50ccfe58-be3f-47b1-a7da-b8781bd18257\" (UID: \"50ccfe58-be3f-47b1-a7da-b8781bd18257\") "
Nov 02 15:00:13.658395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.657787    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle" pod "ea277ef9-22d8-4415-8225-a7c14d5e477e" (UID: "ea277ef9-22d8-4415-8225-a7c14d5e477e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.658395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.658085    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle" pod "50ccfe58-be3f-47b1-a7da-b8781bd18257" (UID: "50ccfe58-be3f-47b1-a7da-b8781bd18257"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.658395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.658167    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.658743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.658720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle" pod "47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" (UID: "47e8b8ec-bd41-43a6-8e01-1b56e7f2d911"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.659776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.659733    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b" pod "ea277ef9-22d8-4415-8225-a7c14d5e477e" (UID: "ea277ef9-22d8-4415-8225-a7c14d5e477e"). InnerVolumeSpecName "kube-api-access-2fv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:13.660572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.660466    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle" pod "7adcaa6c-1e09-4b5d-b285-e4041ae58644" (UID: "7adcaa6c-1e09-4b5d-b285-e4041ae58644"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.661062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.661034    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq" pod "50ccfe58-be3f-47b1-a7da-b8781bd18257" (UID: "50ccfe58-be3f-47b1-a7da-b8781bd18257"). InnerVolumeSpecName "kube-api-access-566tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:13.661350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.661326    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4" pod "47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" (UID: "47e8b8ec-bd41-43a6-8e01-1b56e7f2d911"). InnerVolumeSpecName "kube-api-access-9fcs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:13.662710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.662685    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn" pod "7adcaa6c-1e09-4b5d-b285-e4041ae58644" (UID: "7adcaa6c-1e09-4b5d-b285-e4041ae58644"). InnerVolumeSpecName "kube-api-access-fhxbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:13.664035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.664010    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util" pod "50ccfe58-be3f-47b1-a7da-b8781bd18257" (UID: "50ccfe58-be3f-47b1-a7da-b8781bd18257"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.664264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.664247    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util" pod "7adcaa6c-1e09-4b5d-b285-e4041ae58644" (UID: "7adcaa6c-1e09-4b5d-b285-e4041ae58644"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.664486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.664466    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util" pod "ea277ef9-22d8-4415-8225-a7c14d5e477e" (UID: "ea277ef9-22d8-4415-8225-a7c14d5e477e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.665867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.665844    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util" pod "47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" (UID: "47e8b8ec-bd41-43a6-8e01-1b56e7f2d911"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:13.758901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758881    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.758901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758901    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcs4\" (UniqueName: \"kubernetes.io/projected/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-kube-api-access-9fcs4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758913    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758923    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fv2b\" (UniqueName: \"kubernetes.io/projected/ea277ef9-22d8-4415-8225-a7c14d5e477e-kube-api-access-2fv2b\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758931    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8b8ec-bd41-43a6-8e01-1b56e7f2d911-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758939    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758946    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxbn\" (UniqueName: \"kubernetes.io/projected/7adcaa6c-1e09-4b5d-b285-e4041ae58644-kube-api-access-fhxbn\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758953    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7adcaa6c-1e09-4b5d-b285-e4041ae58644-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758962    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ccfe58-be3f-47b1-a7da-b8781bd18257-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758969    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566tq\" (UniqueName: \"kubernetes.io/projected/50ccfe58-be3f-47b1-a7da-b8781bd18257-kube-api-access-566tq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:13.759005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:13.758977    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea277ef9-22d8-4415-8225-a7c14d5e477e-util\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:00:14.365487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.365454    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7" event={"ID":"47e8b8ec-bd41-43a6-8e01-1b56e7f2d911","Type":"ContainerDied","Data":"8c3f69bc1f1c1eaff787ea6d10ed05bfcad5d271f93cd916d518bbb441937227"}
Nov 02 15:00:14.365887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.365505    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3f69bc1f1c1eaff787ea6d10ed05bfcad5d271f93cd916d518bbb441937227"
Nov 02 15:00:14.365887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.365468    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64466vq7"
Nov 02 15:00:14.367867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.367524    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p"
Nov 02 15:00:14.367867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.367740    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75kc2p" event={"ID":"ea277ef9-22d8-4415-8225-a7c14d5e477e","Type":"ContainerDied","Data":"bff272e3b002c82648d7c163bcfd052c111c01064b8753dec4f12660bce0b2db"}
Nov 02 15:00:14.367867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.367791    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff272e3b002c82648d7c163bcfd052c111c01064b8753dec4f12660bce0b2db"
Nov 02 15:00:14.369709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.369691    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n"
Nov 02 15:00:14.369709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.369708    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahls9n" event={"ID":"50ccfe58-be3f-47b1-a7da-b8781bd18257","Type":"ContainerDied","Data":"54805bc58d2d115ef6f74be451bd7b68bf23b74b43b2f01232754c3d06ab91ad"}
Nov 02 15:00:14.369866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.369749    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54805bc58d2d115ef6f74be451bd7b68bf23b74b43b2f01232754c3d06ab91ad"
Nov 02 15:00:14.371843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.371824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp" event={"ID":"7adcaa6c-1e09-4b5d-b285-e4041ae58644","Type":"ContainerDied","Data":"5d6bc01988fe9c94c9003495b8fcb013a24ccc20db340992da74778ae19501b5"}
Nov 02 15:00:14.371925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.371855    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pp8gp"
Nov 02 15:00:14.371972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:14.371856    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6bc01988fe9c94c9003495b8fcb013a24ccc20db340992da74778ae19501b5"
Nov 02 15:00:22.144130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144100    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"]
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144411    5041 cpu_manager.go:414] "containerMap: removing container" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144422    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144436    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144441    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144449    5041 cpu_manager.go:414] "containerMap: removing container" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144454    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144461    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144466    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144477    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144482    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144488    5041 cpu_manager.go:414] "containerMap: removing container" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144494    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144500    5041 cpu_manager.go:414] "containerMap: removing container" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144505    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144510    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144514    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144522    5041 cpu_manager.go:414] "containerMap: removing container" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144527    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="util"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144537    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144541    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144548    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144554    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144560    5041 cpu_manager.go:414] "containerMap: removing container" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144564    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144574    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144578    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144585    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144589    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144597    5041 cpu_manager.go:414] "containerMap: removing container" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144602    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="pull"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144608    5041 cpu_manager.go:414] "containerMap: removing container" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144613    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="extract"
Nov 02 15:00:22.144600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144621    5041 cpu_manager.go:414] "containerMap: removing container" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="pull"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144625    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="pull"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144634    5041 cpu_manager.go:414] "containerMap: removing container" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="util"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144639    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="util"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144645    5041 cpu_manager.go:414] "containerMap: removing container" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="util"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144649    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="util"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144655    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="util"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144659    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="util"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144665    5041 cpu_manager.go:414] "containerMap: removing container" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144669    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144718    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e8b8ec-bd41-43a6-8e01-1b56e7f2d911" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144727    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="241b0915-97ee-43ac-9f28-01d977ccc050" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144743    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adcaa6c-1e09-4b5d-b285-e4041ae58644" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144749    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ccfe58-be3f-47b1-a7da-b8781bd18257" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144756    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea277ef9-22d8-4415-8225-a7c14d5e477e" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144762    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="daebe4d4-9dc7-4e66-b35c-7b2ef4bd704e" containerName="extract"
Nov 02 15:00:22.145581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.144768    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03a1754-2282-4091-b7dd-c7ee97f6153a" containerName="extract"
Nov 02 15:00:22.147374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.147348    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.161778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.161756    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 15:00:22.171054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.171037    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-d2wzm\""
Nov 02 15:00:22.171120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.171037    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 15:00:22.173686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.173667    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"]
Nov 02 15:00:22.219743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.219717    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18ab55a4-83a0-4e4d-b598-50a4eb9e5a97-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2\" (UID: \"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.219837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.219756    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tgf\" (UniqueName: \"kubernetes.io/projected/18ab55a4-83a0-4e4d-b598-50a4eb9e5a97-kube-api-access-b8tgf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2\" (UID: \"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.321124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.321100    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18ab55a4-83a0-4e4d-b598-50a4eb9e5a97-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2\" (UID: \"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.321209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.321132    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tgf\" (UniqueName: \"kubernetes.io/projected/18ab55a4-83a0-4e4d-b598-50a4eb9e5a97-kube-api-access-b8tgf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2\" (UID: \"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.321537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.321519    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18ab55a4-83a0-4e4d-b598-50a4eb9e5a97-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2\" (UID: \"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.349034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.349009    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tgf\" (UniqueName: \"kubernetes.io/projected/18ab55a4-83a0-4e4d-b598-50a4eb9e5a97-kube-api-access-b8tgf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2\" (UID: \"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.458511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.458489    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"
Nov 02 15:00:22.793714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:22.793593    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2"]
Nov 02 15:00:22.807269 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:00:22.807240    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ab55a4_83a0_4e4d_b598_50a4eb9e5a97.slice/crio-0ae04fe13c4e4d8b84ee705fdbb0fa9e219e0966333e18255d016fa20a5dfab0 WatchSource:0}: Error finding container 0ae04fe13c4e4d8b84ee705fdbb0fa9e219e0966333e18255d016fa20a5dfab0: Status 404 returned error can't find the container with id 0ae04fe13c4e4d8b84ee705fdbb0fa9e219e0966333e18255d016fa20a5dfab0
Nov 02 15:00:23.415001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:23.414962    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2" event={"ID":"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97","Type":"ContainerStarted","Data":"0ae04fe13c4e4d8b84ee705fdbb0fa9e219e0966333e18255d016fa20a5dfab0"}
Nov 02 15:00:25.601913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.601878    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"]
Nov 02 15:00:25.604833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.604813    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:25.621218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.621186    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 15:00:25.621218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.621186    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:25.628165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.628146    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:25.628245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.628194    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pql6l\""
Nov 02 15:00:25.647202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.647180    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"]
Nov 02 15:00:25.766417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.766323    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq96\" (UniqueName: \"kubernetes.io/projected/ac7862b5-e9cf-48af-8cfa-df6763f0e42e-kube-api-access-fkq96\") pod \"dns-operator-controller-manager-8557cbdb84-jxlbw\" (UID: \"ac7862b5-e9cf-48af-8cfa-df6763f0e42e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:25.866909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.866887    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq96\" (UniqueName: \"kubernetes.io/projected/ac7862b5-e9cf-48af-8cfa-df6763f0e42e-kube-api-access-fkq96\") pod \"dns-operator-controller-manager-8557cbdb84-jxlbw\" (UID: \"ac7862b5-e9cf-48af-8cfa-df6763f0e42e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:25.887822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.887800    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq96\" (UniqueName: \"kubernetes.io/projected/ac7862b5-e9cf-48af-8cfa-df6763f0e42e-kube-api-access-fkq96\") pod \"dns-operator-controller-manager-8557cbdb84-jxlbw\" (UID: \"ac7862b5-e9cf-48af-8cfa-df6763f0e42e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:25.917622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:25.917599    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:26.225547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.225518    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"]
Nov 02 15:00:26.227070 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:00:26.227034    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7862b5_e9cf_48af_8cfa_df6763f0e42e.slice/crio-82859d457b364a9c7dc7a680b50cdc03ac481cc5e22bc7cd49c39815627fc900 WatchSource:0}: Error finding container 82859d457b364a9c7dc7a680b50cdc03ac481cc5e22bc7cd49c39815627fc900: Status 404 returned error can't find the container with id 82859d457b364a9c7dc7a680b50cdc03ac481cc5e22bc7cd49c39815627fc900
Nov 02 15:00:26.431883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.431841    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2" event={"ID":"18ab55a4-83a0-4e4d-b598-50a4eb9e5a97","Type":"ContainerStarted","Data":"32bf7ff61bab51ce65bd537da80d1a9d4891b9f984ffc9958e56a49f5d605df9"}
Nov 02 15:00:26.433576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.433548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw" event={"ID":"ac7862b5-e9cf-48af-8cfa-df6763f0e42e","Type":"ContainerStarted","Data":"82859d457b364a9c7dc7a680b50cdc03ac481cc5e22bc7cd49c39815627fc900"}
Nov 02 15:00:26.476299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.476196    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlmk2" podStartSLOduration=1.778505964 podStartE2EDuration="4.476182732s" totalImagesPullingTime="2.697676768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:26.474574822 +0000 UTC m=+278.732644273" watchObservedRunningTime="2026-11-02 15:00:26.476182732 +0000 UTC m=+278.734252172"
Nov 02 15:00:26.898120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.897978    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"]
Nov 02 15:00:26.901094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.901074    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:26.916879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.916859    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 15:00:26.916993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.916967    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 15:00:26.917056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.917045    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xmjrf\""
Nov 02 15:00:26.926230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:26.926206    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"]
Nov 02 15:00:27.079933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.079896    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ctwkf\" (UID: \"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.080433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.079950    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md257\" (UniqueName: \"kubernetes.io/projected/377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8-kube-api-access-md257\") pod \"cert-manager-webhook-5dbbbf75cc-ctwkf\" (UID: \"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.181191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.181154    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ctwkf\" (UID: \"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.181191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.181197    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md257\" (UniqueName: \"kubernetes.io/projected/377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8-kube-api-access-md257\") pod \"cert-manager-webhook-5dbbbf75cc-ctwkf\" (UID: \"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.213788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.213761    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ctwkf\" (UID: \"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.214817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.214796    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md257\" (UniqueName: \"kubernetes.io/projected/377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8-kube-api-access-md257\") pod \"cert-manager-webhook-5dbbbf75cc-ctwkf\" (UID: \"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.223538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.223509    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:27.564737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:27.564652    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"]
Nov 02 15:00:27.578529 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:00:27.568883    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod377b7b4b_5d6b_45ed_96c2_2dd43cd58dc8.slice/crio-481848f921a75fa6e59c1f0dc32cdfc160c0be6c4ed12506ddae90b60fc8b98c WatchSource:0}: Error finding container 481848f921a75fa6e59c1f0dc32cdfc160c0be6c4ed12506ddae90b60fc8b98c: Status 404 returned error can't find the container with id 481848f921a75fa6e59c1f0dc32cdfc160c0be6c4ed12506ddae90b60fc8b98c
Nov 02 15:00:28.445114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:28.445082    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf" event={"ID":"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8","Type":"ContainerStarted","Data":"481848f921a75fa6e59c1f0dc32cdfc160c0be6c4ed12506ddae90b60fc8b98c"}
Nov 02 15:00:29.452519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:29.452473    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw" event={"ID":"ac7862b5-e9cf-48af-8cfa-df6763f0e42e","Type":"ContainerStarted","Data":"b2680dae8e64d3fdfd334b9a564143d4a7020b6e53dded28689d9a9cd1e24097"}
Nov 02 15:00:29.452941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:29.452630    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:29.501955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:29.501886    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw" podStartSLOduration=1.8768907449999999 podStartE2EDuration="4.50187018s" totalImagesPullingTime="2.624979435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:29.499813692 +0000 UTC m=+281.757883143" watchObservedRunningTime="2026-11-02 15:00:29.50187018 +0000 UTC m=+281.759939629"
Nov 02 15:00:31.463821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:31.463782    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf" event={"ID":"377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8","Type":"ContainerStarted","Data":"9a896830d096af9ab5d4e226d706b4150c1be850aaa7807d003eeba3e74c885d"}
Nov 02 15:00:31.464237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:31.463833    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:31.503594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:31.503545    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf" podStartSLOduration=2.518759883 podStartE2EDuration="5.503530061s" totalImagesPullingTime="2.984770178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:31.501919426 +0000 UTC m=+283.759988877" watchObservedRunningTime="2026-11-02 15:00:31.503530061 +0000 UTC m=+283.761599512"
Nov 02 15:00:35.425009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.424972    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"]
Nov 02 15:00:35.428975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.428956    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.446700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.446683    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 15:00:35.450952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.450933    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhs44\" (UniqueName: \"kubernetes.io/projected/8d9defe8-b9ce-46d4-aa39-321322161d35-kube-api-access-hhs44\") pod \"jobset-operator-674cd59dc5-9h4zk\" (UID: \"8d9defe8-b9ce-46d4-aa39-321322161d35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.451017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.451000    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d9defe8-b9ce-46d4-aa39-321322161d35-tmp\") pod \"jobset-operator-674cd59dc5-9h4zk\" (UID: \"8d9defe8-b9ce-46d4-aa39-321322161d35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.456577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.456555    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 15:00:35.458246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.458226    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"]
Nov 02 15:00:35.478311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.478294    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-6cs5s\""
Nov 02 15:00:35.552184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.552163    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d9defe8-b9ce-46d4-aa39-321322161d35-tmp\") pod \"jobset-operator-674cd59dc5-9h4zk\" (UID: \"8d9defe8-b9ce-46d4-aa39-321322161d35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.552287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.552197    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhs44\" (UniqueName: \"kubernetes.io/projected/8d9defe8-b9ce-46d4-aa39-321322161d35-kube-api-access-hhs44\") pod \"jobset-operator-674cd59dc5-9h4zk\" (UID: \"8d9defe8-b9ce-46d4-aa39-321322161d35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.552538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.552521    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d9defe8-b9ce-46d4-aa39-321322161d35-tmp\") pod \"jobset-operator-674cd59dc5-9h4zk\" (UID: \"8d9defe8-b9ce-46d4-aa39-321322161d35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.573232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.573209    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhs44\" (UniqueName: \"kubernetes.io/projected/8d9defe8-b9ce-46d4-aa39-321322161d35-kube-api-access-hhs44\") pod \"jobset-operator-674cd59dc5-9h4zk\" (UID: \"8d9defe8-b9ce-46d4-aa39-321322161d35\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:35.739775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:35.739704    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"
Nov 02 15:00:36.053246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:36.053157    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk"]
Nov 02 15:00:36.057415 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:00:36.057387    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d9defe8_b9ce_46d4_aa39_321322161d35.slice/crio-d0c92ba57f47b65b48db173feb7152712f741bc12dc61b50fbc667f543d03f92 WatchSource:0}: Error finding container d0c92ba57f47b65b48db173feb7152712f741bc12dc61b50fbc667f543d03f92: Status 404 returned error can't find the container with id d0c92ba57f47b65b48db173feb7152712f741bc12dc61b50fbc667f543d03f92
Nov 02 15:00:36.489027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:36.488982    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk" event={"ID":"8d9defe8-b9ce-46d4-aa39-321322161d35","Type":"ContainerStarted","Data":"d0c92ba57f47b65b48db173feb7152712f741bc12dc61b50fbc667f543d03f92"}
Nov 02 15:00:36.767759 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:00:36.767682    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:00:37.471543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:37.471509    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf"
Nov 02 15:00:38.501813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:38.501773    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk" event={"ID":"8d9defe8-b9ce-46d4-aa39-321322161d35","Type":"ContainerStarted","Data":"f813b2611c5294a3874cf9a0faa5c38ef5fed7912bc08fa7dbbb80faa2315712"}
Nov 02 15:00:38.543637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:38.543589    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9h4zk" podStartSLOduration=1.218571542 podStartE2EDuration="3.543575934s" totalImagesPullingTime="2.325004392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:38.542139715 +0000 UTC m=+290.800209165" watchObservedRunningTime="2026-11-02 15:00:38.543575934 +0000 UTC m=+290.801645384"
Nov 02 15:00:39.894314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.894283    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"]
Nov 02 15:00:39.897798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.897782    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:39.936927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.936901    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"]
Nov 02 15:00:39.986013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.985981    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:39.986164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.986032    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d173953-7005-45b8-8217-c8fabad212a4-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:39.986164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.986099    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d173953-7005-45b8-8217-c8fabad212a4-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:39.986164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.986156    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:39.986288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.986174    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:39.986288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:39.986202    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxd5\" (UniqueName: \"kubernetes.io/projected/6d173953-7005-45b8-8217-c8fabad212a4-kube-api-access-vwxd5\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.003803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.003777    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 15:00:40.004155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.004134    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46gbv\""
Nov 02 15:00:40.004155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.004151    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 15:00:40.004277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.004189    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 15:00:40.006707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.006693    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:40.006790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.006779    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:40.087514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087454    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.087514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087492    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d173953-7005-45b8-8217-c8fabad212a4-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.087514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d173953-7005-45b8-8217-c8fabad212a4-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.087704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087538    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.087704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087563    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.087704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087691    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxd5\" (UniqueName: \"kubernetes.io/projected/6d173953-7005-45b8-8217-c8fabad212a4-kube-api-access-vwxd5\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.087980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.087958    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d173953-7005-45b8-8217-c8fabad212a4-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.089828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.089803    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d173953-7005-45b8-8217-c8fabad212a4-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.090108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.090091    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.090169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.090146    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.090207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.090189    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d173953-7005-45b8-8217-c8fabad212a4-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.130099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.130076    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxd5\" (UniqueName: \"kubernetes.io/projected/6d173953-7005-45b8-8217-c8fabad212a4-kube-api-access-vwxd5\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-qmwlr\" (UID: \"6d173953-7005-45b8-8217-c8fabad212a4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.213918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.213896    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:40.460278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.460256    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jxlbw"
Nov 02 15:00:40.565424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:40.565398    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"]
Nov 02 15:00:40.567208 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:00:40.567177    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d173953_7005_45b8_8217_c8fabad212a4.slice/crio-e341c8764a193e6d7ec901ffbfc1cee99e9b166030cc283220793ca2179a479f WatchSource:0}: Error finding container e341c8764a193e6d7ec901ffbfc1cee99e9b166030cc283220793ca2179a479f: Status 404 returned error can't find the container with id e341c8764a193e6d7ec901ffbfc1cee99e9b166030cc283220793ca2179a479f
Nov 02 15:00:41.188057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.188025    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-8xljz"]
Nov 02 15:00:41.191799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.191778    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.196403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.196351    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f0e02ab-aede-4d58-967b-a70ae3480c57-bound-sa-token\") pod \"cert-manager-597b95cb84-8xljz\" (UID: \"3f0e02ab-aede-4d58-967b-a70ae3480c57\") " pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.196520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.196478    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bl8k\" (UniqueName: \"kubernetes.io/projected/3f0e02ab-aede-4d58-967b-a70ae3480c57-kube-api-access-8bl8k\") pod \"cert-manager-597b95cb84-8xljz\" (UID: \"3f0e02ab-aede-4d58-967b-a70ae3480c57\") " pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.209273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.209256    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nlxc8\""
Nov 02 15:00:41.219889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.219843    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8xljz"]
Nov 02 15:00:41.296938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.296908    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f0e02ab-aede-4d58-967b-a70ae3480c57-bound-sa-token\") pod \"cert-manager-597b95cb84-8xljz\" (UID: \"3f0e02ab-aede-4d58-967b-a70ae3480c57\") " pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.297063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.296985    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bl8k\" (UniqueName: \"kubernetes.io/projected/3f0e02ab-aede-4d58-967b-a70ae3480c57-kube-api-access-8bl8k\") pod \"cert-manager-597b95cb84-8xljz\" (UID: \"3f0e02ab-aede-4d58-967b-a70ae3480c57\") " pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.324463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.324437    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bl8k\" (UniqueName: \"kubernetes.io/projected/3f0e02ab-aede-4d58-967b-a70ae3480c57-kube-api-access-8bl8k\") pod \"cert-manager-597b95cb84-8xljz\" (UID: \"3f0e02ab-aede-4d58-967b-a70ae3480c57\") " pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.331991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.331946    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f0e02ab-aede-4d58-967b-a70ae3480c57-bound-sa-token\") pod \"cert-manager-597b95cb84-8xljz\" (UID: \"3f0e02ab-aede-4d58-967b-a70ae3480c57\") " pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.507403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.507142    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8xljz"
Nov 02 15:00:41.523451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.523389    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr" event={"ID":"6d173953-7005-45b8-8217-c8fabad212a4","Type":"ContainerStarted","Data":"e341c8764a193e6d7ec901ffbfc1cee99e9b166030cc283220793ca2179a479f"}
Nov 02 15:00:41.839215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:41.839123    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8xljz"]
Nov 02 15:00:41.843189 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:00:41.843159    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f0e02ab_aede_4d58_967b_a70ae3480c57.slice/crio-15b38264a2fabb21271bbc2019c7aef42ee308139b9f99bc7c9255183a0c6a61 WatchSource:0}: Error finding container 15b38264a2fabb21271bbc2019c7aef42ee308139b9f99bc7c9255183a0c6a61: Status 404 returned error can't find the container with id 15b38264a2fabb21271bbc2019c7aef42ee308139b9f99bc7c9255183a0c6a61
Nov 02 15:00:42.530800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:42.530766    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8xljz" event={"ID":"3f0e02ab-aede-4d58-967b-a70ae3480c57","Type":"ContainerStarted","Data":"27333f8504c301708c4b1eec13c0e60fb7a1f1a59460e81c15bd84269f624816"}
Nov 02 15:00:42.531277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:42.530811    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8xljz" event={"ID":"3f0e02ab-aede-4d58-967b-a70ae3480c57","Type":"ContainerStarted","Data":"15b38264a2fabb21271bbc2019c7aef42ee308139b9f99bc7c9255183a0c6a61"}
Nov 02 15:00:42.570068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:42.570014    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-8xljz" podStartSLOduration=1.5699974380000001 podStartE2EDuration="1.569997438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:42.567675381 +0000 UTC m=+294.825744828" watchObservedRunningTime="2026-11-02 15:00:42.569997438 +0000 UTC m=+294.828066887"
Nov 02 15:00:44.542732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:44.542705    5041 generic.go:356] "Generic (PLEG): container finished" podID="6d173953-7005-45b8-8217-c8fabad212a4" containerID="d47ba0b38d99a7933ada4909291724094ff512cd6f88e1aab3d77e8b89e1870a" exitCode=0
Nov 02 15:00:44.543087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:44.542793    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr" event={"ID":"6d173953-7005-45b8-8217-c8fabad212a4","Type":"ContainerDied","Data":"d47ba0b38d99a7933ada4909291724094ff512cd6f88e1aab3d77e8b89e1870a"}
Nov 02 15:00:45.551086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:45.551021    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr" event={"ID":"6d173953-7005-45b8-8217-c8fabad212a4","Type":"ContainerStarted","Data":"d04bb6b7deaae8a84de6850534431eda45f61cea1b08b30fffd35155148dc4fa"}
Nov 02 15:00:45.551594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:45.551255    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:47.918725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:47.918693    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:00:47.919225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:47.918700    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:00:56.558221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:56.558190    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr"
Nov 02 15:00:56.652238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:00:56.652189    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-qmwlr" podStartSLOduration=13.805095115 podStartE2EDuration="17.652177243s" totalImagesPullingTime="3.362600131s" totalInitContainerRuntime="484.481997ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:45.749031029 +0000 UTC m=+298.007100480" watchObservedRunningTime="2026-11-02 15:00:56.652177243 +0000 UTC m=+308.910246692"
Nov 02 15:01:06.767336 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:01:06.767258    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:01:19.138258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.138219    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77f469495f-npwt9"]
Nov 02 15:01:19.141734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.141709    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.175210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.175184    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 15:01:19.175210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.175201    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m9qjk\""
Nov 02 15:01:19.184566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.184537    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 15:01:19.187421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.187393    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77f469495f-npwt9"]
Nov 02 15:01:19.288614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.288580    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrhr\" (UniqueName: \"kubernetes.io/projected/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-kube-api-access-kmrhr\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.288778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.288635    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-tls-certs\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.288778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.288712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-tmp\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.390165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.390082    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-tls-certs\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.390165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.390128    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-tmp\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.390422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.390209    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrhr\" (UniqueName: \"kubernetes.io/projected/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-kube-api-access-kmrhr\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.392447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.392414    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-tmp\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.392779 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.392757    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-tls-certs\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.431733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.431700    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrhr\" (UniqueName: \"kubernetes.io/projected/9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf-kube-api-access-kmrhr\") pod \"kube-auth-proxy-77f469495f-npwt9\" (UID: \"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.452873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.452847    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9"
Nov 02 15:01:19.767524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.767496    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77f469495f-npwt9"]
Nov 02 15:01:19.770009 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:01:19.769983    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed68aba_0ac0_4d2a_9ff1_4b3302f13abf.slice/crio-93f7dbd2982b471040a27017d40d39132a95e5231ccf7e68cb104d4b8c6e906d WatchSource:0}: Error finding container 93f7dbd2982b471040a27017d40d39132a95e5231ccf7e68cb104d4b8c6e906d: Status 404 returned error can't find the container with id 93f7dbd2982b471040a27017d40d39132a95e5231ccf7e68cb104d4b8c6e906d
Nov 02 15:01:19.772439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:19.772423    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:01:20.726072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:20.726031    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9" event={"ID":"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf","Type":"ContainerStarted","Data":"93f7dbd2982b471040a27017d40d39132a95e5231ccf7e68cb104d4b8c6e906d"}
Nov 02 15:01:22.549139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.549068    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hqrp8"]
Nov 02 15:01:22.551991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.551965    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:22.576761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.576739    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hqrp8"]
Nov 02 15:01:22.578124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.578103    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsl2r\""
Nov 02 15:01:22.618963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.618939    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfb8\" (UniqueName: \"kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8\") pod \"authorino-57f7f8659f-hqrp8\" (UID: \"daff0356-9cf5-43d3-9c7d-aef3db078bde\") " pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:22.719335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.719312    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfb8\" (UniqueName: \"kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8\") pod \"authorino-57f7f8659f-hqrp8\" (UID: \"daff0356-9cf5-43d3-9c7d-aef3db078bde\") " pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:22.739110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.739078    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9" event={"ID":"9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf","Type":"ContainerStarted","Data":"8faf0a3dcd72dc487b36f9a6ce42ad3e236301c45637923bcb402a2f863ac8de"}
Nov 02 15:01:22.741724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.741703    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfb8\" (UniqueName: \"kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8\") pod \"authorino-57f7f8659f-hqrp8\" (UID: \"daff0356-9cf5-43d3-9c7d-aef3db078bde\") " pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:22.786666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.786619    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77f469495f-npwt9" podStartSLOduration=1.354140624 podStartE2EDuration="3.786604324s" totalImagesPullingTime="2.4324637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:22.785796 +0000 UTC m=+335.043865461" watchObservedRunningTime="2026-11-02 15:01:22.786604324 +0000 UTC m=+335.044673778"
Nov 02 15:01:22.863218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:22.863145    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:23.167625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.167585    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hqrp8"]
Nov 02 15:01:23.169033 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:01:23.169001    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaff0356_9cf5_43d3_9c7d_aef3db078bde.slice/crio-4d8d337589888e3402d2184b3fd49584986cf20a537bc3bdb76c8f2c87aadd3b WatchSource:0}: Error finding container 4d8d337589888e3402d2184b3fd49584986cf20a537bc3bdb76c8f2c87aadd3b: Status 404 returned error can't find the container with id 4d8d337589888e3402d2184b3fd49584986cf20a537bc3bdb76c8f2c87aadd3b
Nov 02 15:01:23.291514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.291486    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kg6c2"]
Nov 02 15:01:23.294969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.294944    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:01:23.318910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.318888    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kg6c2"]
Nov 02 15:01:23.424410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.424350    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkc4\" (UniqueName: \"kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4\") pod \"authorino-56c6c9ddd5-kg6c2\" (UID: \"6644b5bc-2d25-473c-9768-33af5157561f\") " pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:01:23.525255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.525226    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkc4\" (UniqueName: \"kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4\") pod \"authorino-56c6c9ddd5-kg6c2\" (UID: \"6644b5bc-2d25-473c-9768-33af5157561f\") " pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:01:23.582057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.582027    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkc4\" (UniqueName: \"kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4\") pod \"authorino-56c6c9ddd5-kg6c2\" (UID: \"6644b5bc-2d25-473c-9768-33af5157561f\") " pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:01:23.606795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.606773    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:01:23.745324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.745258    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hqrp8" event={"ID":"daff0356-9cf5-43d3-9c7d-aef3db078bde","Type":"ContainerStarted","Data":"4d8d337589888e3402d2184b3fd49584986cf20a537bc3bdb76c8f2c87aadd3b"}
Nov 02 15:01:23.899094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:23.899067    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kg6c2"]
Nov 02 15:01:23.900771 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:01:23.900740    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6644b5bc_2d25_473c_9768_33af5157561f.slice/crio-5caa4caf46b21b1d6e55ec516d90970f7e7fcd9f1a98b5bab14b0d03b2108394 WatchSource:0}: Error finding container 5caa4caf46b21b1d6e55ec516d90970f7e7fcd9f1a98b5bab14b0d03b2108394: Status 404 returned error can't find the container with id 5caa4caf46b21b1d6e55ec516d90970f7e7fcd9f1a98b5bab14b0d03b2108394
Nov 02 15:01:24.753299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:24.753259    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2" event={"ID":"6644b5bc-2d25-473c-9768-33af5157561f","Type":"ContainerStarted","Data":"5caa4caf46b21b1d6e55ec516d90970f7e7fcd9f1a98b5bab14b0d03b2108394"}
Nov 02 15:01:26.765955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:26.765919    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hqrp8" event={"ID":"daff0356-9cf5-43d3-9c7d-aef3db078bde","Type":"ContainerStarted","Data":"3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef"}
Nov 02 15:01:26.767329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:26.767299    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2" event={"ID":"6644b5bc-2d25-473c-9768-33af5157561f","Type":"ContainerStarted","Data":"702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a"}
Nov 02 15:01:26.804357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:26.804312    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-hqrp8" podStartSLOduration=2.095545786 podStartE2EDuration="4.804299627s" totalImagesPullingTime="2.708753841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:26.803124771 +0000 UTC m=+339.061194222" watchObservedRunningTime="2026-11-02 15:01:26.804299627 +0000 UTC m=+339.062369076"
Nov 02 15:01:26.843510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:26.843467    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2" podStartSLOduration=1.795206107 podStartE2EDuration="3.843455712s" totalImagesPullingTime="2.048249605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:26.84194181 +0000 UTC m=+339.100011261" watchObservedRunningTime="2026-11-02 15:01:26.843455712 +0000 UTC m=+339.101525162"
Nov 02 15:01:26.892184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:26.892153    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hqrp8"]
Nov 02 15:01:28.326109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.326074    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"]
Nov 02 15:01:28.328570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.328552    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.352800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.352776    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 15:01:28.352800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.352776    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 15:01:28.373099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.373077    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-7ls6t\""
Nov 02 15:01:28.373224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.373209    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 15:01:28.444007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.443984    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"]
Nov 02 15:01:28.467533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.467508    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/717f7a98-82e5-4412-a803-dd3b77fe1c4a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.467657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.467546    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/717f7a98-82e5-4412-a803-dd3b77fe1c4a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.467657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.467577    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2tc\" (UniqueName: \"kubernetes.io/projected/717f7a98-82e5-4412-a803-dd3b77fe1c4a-kube-api-access-2v2tc\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.467788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.467683    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/717f7a98-82e5-4412-a803-dd3b77fe1c4a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.568509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.568486    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/717f7a98-82e5-4412-a803-dd3b77fe1c4a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.568598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.568524    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2tc\" (UniqueName: \"kubernetes.io/projected/717f7a98-82e5-4412-a803-dd3b77fe1c4a-kube-api-access-2v2tc\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.568598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.568549    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/717f7a98-82e5-4412-a803-dd3b77fe1c4a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.568675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.568625    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/717f7a98-82e5-4412-a803-dd3b77fe1c4a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.568847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.568830    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/717f7a98-82e5-4412-a803-dd3b77fe1c4a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.568941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.568925    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/717f7a98-82e5-4412-a803-dd3b77fe1c4a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.569332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.569316    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/717f7a98-82e5-4412-a803-dd3b77fe1c4a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.589968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.589911    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2tc\" (UniqueName: \"kubernetes.io/projected/717f7a98-82e5-4412-a803-dd3b77fe1c4a-kube-api-access-2v2tc\") pod \"kserve-module-controller-manager-7566cf5fc8-wbgvb\" (UID: \"717f7a98-82e5-4412-a803-dd3b77fe1c4a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.640423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.640401    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:28.777169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.777126    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-hqrp8" podUID="daff0356-9cf5-43d3-9c7d-aef3db078bde" containerName="authorino" containerID="cri-o://3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef" gracePeriod=30
Nov 02 15:01:28.950359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:28.950330    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"]
Nov 02 15:01:28.954119 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:01:28.954085    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717f7a98_82e5_4412_a803_dd3b77fe1c4a.slice/crio-33f56c6c14582afe3353a53e7df3109929ff9e9e05b5e69df6522199426546fc WatchSource:0}: Error finding container 33f56c6c14582afe3353a53e7df3109929ff9e9e05b5e69df6522199426546fc: Status 404 returned error can't find the container with id 33f56c6c14582afe3353a53e7df3109929ff9e9e05b5e69df6522199426546fc
Nov 02 15:01:29.011850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.011827    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:29.178047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.178017    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8\" (UniqueName: \"kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8\") pod \"daff0356-9cf5-43d3-9c7d-aef3db078bde\" (UID: \"daff0356-9cf5-43d3-9c7d-aef3db078bde\") "
Nov 02 15:01:29.180079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.180050    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8" pod "daff0356-9cf5-43d3-9c7d-aef3db078bde" (UID: "daff0356-9cf5-43d3-9c7d-aef3db078bde"). InnerVolumeSpecName "kube-api-access-9nfb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:29.278698 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.278665    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfb8\" (UniqueName: \"kubernetes.io/projected/daff0356-9cf5-43d3-9c7d-aef3db078bde-kube-api-access-9nfb8\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:01:29.785408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.785346    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb" event={"ID":"717f7a98-82e5-4412-a803-dd3b77fe1c4a","Type":"ContainerStarted","Data":"33f56c6c14582afe3353a53e7df3109929ff9e9e05b5e69df6522199426546fc"}
Nov 02 15:01:29.786780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.786752    5041 generic.go:356] "Generic (PLEG): container finished" podID="daff0356-9cf5-43d3-9c7d-aef3db078bde" containerID="3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef" exitCode=0
Nov 02 15:01:29.786919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.786795    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hqrp8"
Nov 02 15:01:29.786919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.786833    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hqrp8" event={"ID":"daff0356-9cf5-43d3-9c7d-aef3db078bde","Type":"ContainerDied","Data":"3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef"}
Nov 02 15:01:29.786919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.786878    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hqrp8" event={"ID":"daff0356-9cf5-43d3-9c7d-aef3db078bde","Type":"ContainerDied","Data":"4d8d337589888e3402d2184b3fd49584986cf20a537bc3bdb76c8f2c87aadd3b"}
Nov 02 15:01:29.786919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.786901    5041 scope.go:122] "RemoveContainer" containerID="3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef"
Nov 02 15:01:29.797066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.796981    5041 scope.go:122] "RemoveContainer" containerID="3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef"
Nov 02 15:01:29.797751 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:01:29.797728    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef\": container with ID starting with 3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef not found: ID does not exist" containerID="3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef"
Nov 02 15:01:29.797902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.797766    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef"} err="failed to get container status \"3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef\": rpc error: code = NotFound desc = could not find container \"3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef\": container with ID starting with 3dd0f2a1ab2b5c2f72c1c375071927ac8a15b57e0f9bbac6ad51fd628569dbef not found: ID does not exist"
Nov 02 15:01:29.831419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.831386    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hqrp8"]
Nov 02 15:01:29.835093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.835051    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hqrp8"]
Nov 02 15:01:29.887168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:29.887088    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daff0356-9cf5-43d3-9c7d-aef3db078bde" path="/var/lib/kubelet/pods/daff0356-9cf5-43d3-9c7d-aef3db078bde/volumes"
Nov 02 15:01:30.794524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:30.794484    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb" event={"ID":"717f7a98-82e5-4412-a803-dd3b77fe1c4a","Type":"ContainerStarted","Data":"7aee729195db77eba4b1ab00a470a9230b457e47c51e265828bee618ae4ed01e"}
Nov 02 15:01:30.794919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:30.794599    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:01:30.842790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:30.842694    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb" podStartSLOduration=1.236022927 podStartE2EDuration="2.84267779s" totalImagesPullingTime="1.606654863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:30.841495913 +0000 UTC m=+343.099565362" watchObservedRunningTime="2026-11-02 15:01:30.84267779 +0000 UTC m=+343.100747239"
Nov 02 15:01:36.767088 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:01:36.767056    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:01:51.801240 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:01:51.801209    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wbgvb"
Nov 02 15:02:06.769523 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:02:06.769485    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:02:11.535518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.535485    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-pfkzw"]
Nov 02 15:02:11.536484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.536454    5041 cpu_manager.go:414] "containerMap: removing container" podUID="daff0356-9cf5-43d3-9c7d-aef3db078bde" containerName="authorino"
Nov 02 15:02:11.536484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.536476    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daff0356-9cf5-43d3-9c7d-aef3db078bde" containerName="authorino"
Nov 02 15:02:11.536666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.536637    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="daff0356-9cf5-43d3-9c7d-aef3db078bde" containerName="authorino"
Nov 02 15:02:11.540179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.540156    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.554806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.554787    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 15:02:11.566876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.566836    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 15:02:11.566876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.566870    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 15:02:11.567041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.566967    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kr787\""
Nov 02 15:02:11.569721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.569696    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-pfkzw"]
Nov 02 15:02:11.613664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.613629    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.613828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.613683    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.613828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.613780    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.714983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.714949    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.715169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.715003    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.715169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.715026    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.717589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.717562    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.717712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.717560    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.738884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.738845    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7\") pod \"maas-controller-579cf7c6d-pfkzw\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") " pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.855480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.855343    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:11.873777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.873750    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hvszm"]
Nov 02 15:02:11.877228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.877210    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:11.922298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.922255    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:11.922539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.922463    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjcl\" (UniqueName: \"kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:11.922623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.922573    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:11.928412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:11.928355    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hvszm"]
Nov 02 15:02:12.024030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.023993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.024196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.024056    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjcl\" (UniqueName: \"kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.024196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.024102    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.026637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.026612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.026756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.026659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.046847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.046826    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjcl\" (UniqueName: \"kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl\") pod \"maas-controller-6667f79cdf-hvszm\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") " pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.182453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.182425    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-pfkzw"]
Nov 02 15:02:12.184012 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:02:12.183984    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a3f880_a795_4a89_8974_7e20de3b0a6a.slice/crio-301d98cd38620fbb89ff19873835630a19451d46219f5b08bb2a55f6afea4597 WatchSource:0}: Error finding container 301d98cd38620fbb89ff19873835630a19451d46219f5b08bb2a55f6afea4597: Status 404 returned error can't find the container with id 301d98cd38620fbb89ff19873835630a19451d46219f5b08bb2a55f6afea4597
Nov 02 15:02:12.191966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.191941    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:12.491163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:12.491126    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hvszm"]
Nov 02 15:02:12.492727 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:02:12.492694    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce27d7ab_8df4_47df_8564_a18e220911f6.slice/crio-4f1a9dc29f4ab47a5545dabbaa22d9d0631a2c8a5c960a9320a6209ed62e5b1a WatchSource:0}: Error finding container 4f1a9dc29f4ab47a5545dabbaa22d9d0631a2c8a5c960a9320a6209ed62e5b1a: Status 404 returned error can't find the container with id 4f1a9dc29f4ab47a5545dabbaa22d9d0631a2c8a5c960a9320a6209ed62e5b1a
Nov 02 15:02:13.006243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:13.006167    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-pfkzw" event={"ID":"c0a3f880-a795-4a89-8974-7e20de3b0a6a","Type":"ContainerStarted","Data":"301d98cd38620fbb89ff19873835630a19451d46219f5b08bb2a55f6afea4597"}
Nov 02 15:02:13.007884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:13.007847    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-hvszm" event={"ID":"ce27d7ab-8df4-47df-8564-a18e220911f6","Type":"ContainerStarted","Data":"4f1a9dc29f4ab47a5545dabbaa22d9d0631a2c8a5c960a9320a6209ed62e5b1a"}
Nov 02 15:02:16.028934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:16.028899    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-hvszm" event={"ID":"ce27d7ab-8df4-47df-8564-a18e220911f6","Type":"ContainerStarted","Data":"d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e"}
Nov 02 15:02:16.029420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:16.028962    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:16.030543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:16.030517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-pfkzw" event={"ID":"c0a3f880-a795-4a89-8974-7e20de3b0a6a","Type":"ContainerStarted","Data":"4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba"}
Nov 02 15:02:16.030669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:16.030582    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:16.074304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:16.074258    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6667f79cdf-hvszm" podStartSLOduration=2.279616265 podStartE2EDuration="5.074242906s" totalImagesPullingTime="2.794626641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:16.073516744 +0000 UTC m=+388.331586194" watchObservedRunningTime="2026-11-02 15:02:16.074242906 +0000 UTC m=+388.332312389"
Nov 02 15:02:23.388653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.388595    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-pfkzw" podStartSLOduration=9.392057557 podStartE2EDuration="12.388580531s" totalImagesPullingTime="2.996522974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:16.127040034 +0000 UTC m=+388.385109485" watchObservedRunningTime="2026-11-02 15:02:23.388580531 +0000 UTC m=+395.646649981"
Nov 02 15:02:23.389053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.388960    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"]
Nov 02 15:02:23.394013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.393995    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.408831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.408797    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 15:02:23.420117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.420089    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 15:02:23.420117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.420096    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 15:02:23.420117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.420100    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 15:02:23.420272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.420146    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rcl8t\""
Nov 02 15:02:23.426185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.426166    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"]
Nov 02 15:02:23.527779 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.527747    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.527913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.527783    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blll8\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.527913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.527809    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.527984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.527916    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.527984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.527963    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.528048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.528037    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.528078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.528058    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.628976    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629018    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629038    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629080    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629097    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blll8\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629113    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629133    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.629724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.629693    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.631696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.631664    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.631833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.631815    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.631907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.631878    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.631964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.631930    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.651650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.651598    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.652029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.652007    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blll8\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7t9p\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:23.705540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:23.705516    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:24.018758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:24.018723    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"]
Nov 02 15:02:24.020748 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:02:24.020716    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b13603_aaac_454d_9734_807b6e69ffe7.slice/crio-6c7d2d788bbd1a6cbdefd7e0b3c3b23c3ededcc16e9ade6708736ad3477ebff6 WatchSource:0}: Error finding container 6c7d2d788bbd1a6cbdefd7e0b3c3b23c3ededcc16e9ade6708736ad3477ebff6: Status 404 returned error can't find the container with id 6c7d2d788bbd1a6cbdefd7e0b3c3b23c3ededcc16e9ade6708736ad3477ebff6
Nov 02 15:02:24.091226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:24.091192    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" event={"ID":"b1b13603-aaac-454d-9734-807b6e69ffe7","Type":"ContainerStarted","Data":"6c7d2d788bbd1a6cbdefd7e0b3c3b23c3ededcc16e9ade6708736ad3477ebff6"}
Nov 02 15:02:26.345930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:26.345898    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:02:26.346183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:26.345964    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:02:27.038412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.038378    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:27.038576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.038457    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:02:27.110953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.110920    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" event={"ID":"b1b13603-aaac-454d-9734-807b6e69ffe7","Type":"ContainerStarted","Data":"d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98"}
Nov 02 15:02:27.111094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.110986    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:27.112732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.112651    5041 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-z7t9p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:02:27.112732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.112711    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" podUID="b1b13603-aaac-454d-9734-807b6e69ffe7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:02:27.158411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.158374    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-pfkzw"]
Nov 02 15:02:27.158584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.158546    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-pfkzw" podUID="c0a3f880-a795-4a89-8974-7e20de3b0a6a" containerName="manager" containerID="cri-o://4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba" gracePeriod=30
Nov 02 15:02:27.214797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.214694    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" podStartSLOduration=1.892407425 podStartE2EDuration="4.214671874s" totalImagesPullingTime="2.322264449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:27.169735444 +0000 UTC m=+399.427804895" watchObservedRunningTime="2026-11-02 15:02:27.214671874 +0000 UTC m=+399.472741334"
Nov 02 15:02:27.408434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.408408    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:27.565827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.565800    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs\") pod \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") "
Nov 02 15:02:27.565980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.565847    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert\") pod \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") "
Nov 02 15:02:27.565980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.565910    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7\") pod \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\" (UID: \"c0a3f880-a795-4a89-8974-7e20de3b0a6a\") "
Nov 02 15:02:27.568124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.568090    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert" pod "c0a3f880-a795-4a89-8974-7e20de3b0a6a" (UID: "c0a3f880-a795-4a89-8974-7e20de3b0a6a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:27.568227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.568171    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7" pod "c0a3f880-a795-4a89-8974-7e20de3b0a6a" (UID: "c0a3f880-a795-4a89-8974-7e20de3b0a6a"). InnerVolumeSpecName "kube-api-access-2gjn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:02:27.568227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.568169    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs" pod "c0a3f880-a795-4a89-8974-7e20de3b0a6a" (UID: "c0a3f880-a795-4a89-8974-7e20de3b0a6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:27.666448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.666392    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:02:27.666448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.666415    5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0a3f880-a795-4a89-8974-7e20de3b0a6a-webhook-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:02:27.666448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:27.666424    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/c0a3f880-a795-4a89-8974-7e20de3b0a6a-kube-api-access-2gjn7\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.116917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.116889    5041 generic.go:356] "Generic (PLEG): container finished" podID="c0a3f880-a795-4a89-8974-7e20de3b0a6a" containerID="4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba" exitCode=0
Nov 02 15:02:28.117073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.116937    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-pfkzw"
Nov 02 15:02:28.117073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.116972    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-pfkzw" event={"ID":"c0a3f880-a795-4a89-8974-7e20de3b0a6a","Type":"ContainerDied","Data":"4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba"}
Nov 02 15:02:28.117073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.117026    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-pfkzw" event={"ID":"c0a3f880-a795-4a89-8974-7e20de3b0a6a","Type":"ContainerDied","Data":"301d98cd38620fbb89ff19873835630a19451d46219f5b08bb2a55f6afea4597"}
Nov 02 15:02:28.117073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.117055    5041 scope.go:122] "RemoveContainer" containerID="4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba"
Nov 02 15:02:28.118168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.118155    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:02:28.125825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.125807    5041 scope.go:122] "RemoveContainer" containerID="4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba"
Nov 02 15:02:28.126085 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:02:28.126067    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba\": container with ID starting with 4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba not found: ID does not exist" containerID="4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba"
Nov 02 15:02:28.126144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.126093    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba"} err="failed to get container status \"4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba\": rpc error: code = NotFound desc = could not find container \"4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba\": container with ID starting with 4539a7e64854c6a54d3cca96754de3e31b48cbcd13299883973a5bd34bdd1cba not found: ID does not exist"
Nov 02 15:02:28.225004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.224968    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-pfkzw"]
Nov 02 15:02:28.234733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:28.234702    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-pfkzw"]
Nov 02 15:02:29.886895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:02:29.886826    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a3f880-a795-4a89-8974-7e20de3b0a6a" path="/var/lib/kubelet/pods/c0a3f880-a795-4a89-8974-7e20de3b0a6a/volumes"
Nov 02 15:02:36.768518 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:02:36.768477    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:03:04.562716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:04.562673    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kg6c2"]
Nov 02 15:03:04.563346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:04.562927    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2" podUID="6644b5bc-2d25-473c-9768-33af5157561f" containerName="authorino" containerID="cri-o://702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a" gracePeriod=30
Nov 02 15:03:04.820545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:04.820480    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:03:04.890232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:04.890200    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4\" (UniqueName: \"kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4\") pod \"6644b5bc-2d25-473c-9768-33af5157561f\" (UID: \"6644b5bc-2d25-473c-9768-33af5157561f\") "
Nov 02 15:03:04.892227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:04.892199    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4" pod "6644b5bc-2d25-473c-9768-33af5157561f" (UID: "6644b5bc-2d25-473c-9768-33af5157561f"). InnerVolumeSpecName "kube-api-access-lmkc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:04.991614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:04.991582    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkc4\" (UniqueName: \"kubernetes.io/projected/6644b5bc-2d25-473c-9768-33af5157561f-kube-api-access-lmkc4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:03:05.312543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.312510    5041 generic.go:356] "Generic (PLEG): container finished" podID="6644b5bc-2d25-473c-9768-33af5157561f" containerID="702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a" exitCode=0
Nov 02 15:03:05.312717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.312558    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2" event={"ID":"6644b5bc-2d25-473c-9768-33af5157561f","Type":"ContainerDied","Data":"702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a"}
Nov 02 15:03:05.312717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.312569    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2"
Nov 02 15:03:05.312717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.312589    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kg6c2" event={"ID":"6644b5bc-2d25-473c-9768-33af5157561f","Type":"ContainerDied","Data":"5caa4caf46b21b1d6e55ec516d90970f7e7fcd9f1a98b5bab14b0d03b2108394"}
Nov 02 15:03:05.312717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.312605    5041 scope.go:122] "RemoveContainer" containerID="702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a"
Nov 02 15:03:05.321685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.321665    5041 scope.go:122] "RemoveContainer" containerID="702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a"
Nov 02 15:03:05.321958 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:03:05.321940    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a\": container with ID starting with 702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a not found: ID does not exist" containerID="702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a"
Nov 02 15:03:05.322002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.321968    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a"} err="failed to get container status \"702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a\": rpc error: code = NotFound desc = could not find container \"702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a\": container with ID starting with 702dd1e1abe765a0c3f14454ca7bcb128263f0ac5bfb7eda18fd769642bab05a not found: ID does not exist"
Nov 02 15:03:05.394699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.394672    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kg6c2"]
Nov 02 15:03:05.402744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.402725    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kg6c2"]
Nov 02 15:03:05.885678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:05.885648    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6644b5bc-2d25-473c-9768-33af5157561f" path="/var/lib/kubelet/pods/6644b5bc-2d25-473c-9768-33af5157561f/volumes"
Nov 02 15:03:06.767669 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:03:06.767623    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:03:36.768495 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:03:36.768455    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:03:40.826282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:03:40.826233    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" podUID="15b2f7b139a98a51bb6e53a9a5605774" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:04:06.767019 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:04:06.766948    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:04:27.625825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.625788    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"]
Nov 02 15:04:27.626354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.626334    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6644b5bc-2d25-473c-9768-33af5157561f" containerName="authorino"
Nov 02 15:04:27.626555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.626358    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6644b5bc-2d25-473c-9768-33af5157561f" containerName="authorino"
Nov 02 15:04:27.626555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.626489    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c0a3f880-a795-4a89-8974-7e20de3b0a6a" containerName="manager"
Nov 02 15:04:27.626555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.626498    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a3f880-a795-4a89-8974-7e20de3b0a6a" containerName="manager"
Nov 02 15:04:27.626705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.626619    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6644b5bc-2d25-473c-9768-33af5157561f" containerName="authorino"
Nov 02 15:04:27.626705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.626636    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a3f880-a795-4a89-8974-7e20de3b0a6a" containerName="manager"
Nov 02 15:04:27.628863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.628843    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.654413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.654391    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 15:04:27.654696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.654676    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w4xr6\""
Nov 02 15:04:27.654789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.654756    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 15:04:27.654863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.654845    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 15:04:27.659674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.659654    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"]
Nov 02 15:04:27.755445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.755415    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lphbk\" (UniqueName: \"kubernetes.io/projected/39e554bb-b67f-471a-8990-5fa031445730-kube-api-access-lphbk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.755569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.755451    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.755569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.755481    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e554bb-b67f-471a-8990-5fa031445730-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.755672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.755584    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.755672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.755628    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.755779 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.755671    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.856751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.856722    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e554bb-b67f-471a-8990-5fa031445730-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.856890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.856759    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.856890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.856808    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.856890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.856855    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.857055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.856919    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lphbk\" (UniqueName: \"kubernetes.io/projected/39e554bb-b67f-471a-8990-5fa031445730-kube-api-access-lphbk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.857055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.856979    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.857951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.857929    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.858064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.857974    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.858064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.858050    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.859874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.859855    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e554bb-b67f-471a-8990-5fa031445730-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.860134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.860112    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e554bb-b67f-471a-8990-5fa031445730-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.879545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.879495    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lphbk\" (UniqueName: \"kubernetes.io/projected/39e554bb-b67f-471a-8990-5fa031445730-kube-api-access-lphbk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kbk89\" (UID: \"39e554bb-b67f-471a-8990-5fa031445730\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:27.941926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:27.941902    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:28.241448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:28.241422    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"]
Nov 02 15:04:28.243141 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:04:28.243117    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e554bb_b67f_471a_8990_5fa031445730.slice/crio-8e5bd34dca84964078e659e5e047c6ec9cd96341b92603f2581e56b80f8cdf03 WatchSource:0}: Error finding container 8e5bd34dca84964078e659e5e047c6ec9cd96341b92603f2581e56b80f8cdf03: Status 404 returned error can't find the container with id 8e5bd34dca84964078e659e5e047c6ec9cd96341b92603f2581e56b80f8cdf03
Nov 02 15:04:28.747176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:28.747127    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89" event={"ID":"39e554bb-b67f-471a-8990-5fa031445730","Type":"ContainerStarted","Data":"8e5bd34dca84964078e659e5e047c6ec9cd96341b92603f2581e56b80f8cdf03"}
Nov 02 15:04:29.754416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:29.754359    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89" event={"ID":"39e554bb-b67f-471a-8990-5fa031445730","Type":"ContainerStarted","Data":"ed664d65304364b84bf1b3ad3bfcfe30385d1573c73722f98d29e4eb7b87b3fe"}
Nov 02 15:04:29.800194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:29.800148    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89" podStartSLOduration=1.774427378 podStartE2EDuration="2.800134919s" totalImagesPullingTime="1.025707541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:29.799747411 +0000 UTC m=+522.057816861" watchObservedRunningTime="2026-11-02 15:04:29.800134919 +0000 UTC m=+522.058204368"
Nov 02 15:04:29.943060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:29.943031    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:31.113314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.113242    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"]
Nov 02 15:04:31.116569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.116549    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.135064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.135040    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 15:04:31.145080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.145055    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"]
Nov 02 15:04:31.286553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.286525    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.286683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.286565    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.286683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.286591    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.286769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.286731    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296sc\" (UniqueName: \"kubernetes.io/projected/214f8315-f258-40d4-bb11-bfae94d43ff1-kube-api-access-296sc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.286804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.286766    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/214f8315-f258-40d4-bb11-bfae94d43ff1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.286804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.286789    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388050    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388089    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388111    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388146    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296sc\" (UniqueName: \"kubernetes.io/projected/214f8315-f258-40d4-bb11-bfae94d43ff1-kube-api-access-296sc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388176    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/214f8315-f258-40d4-bb11-bfae94d43ff1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388440    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388557    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.388647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.388612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.390428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.390405    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/214f8315-f258-40d4-bb11-bfae94d43ff1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.390660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.390633    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/214f8315-f258-40d4-bb11-bfae94d43ff1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.412605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.412583    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296sc\" (UniqueName: \"kubernetes.io/projected/214f8315-f258-40d4-bb11-bfae94d43ff1-kube-api-access-296sc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg\" (UID: \"214f8315-f258-40d4-bb11-bfae94d43ff1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.430563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.430539    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:31.728352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.728323    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"]
Nov 02 15:04:31.729636 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:04:31.729609    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214f8315_f258_40d4_bb11_bfae94d43ff1.slice/crio-7bb1d06ee5189f5bb1eaebe2571598d8ed0f2279c814aab840512a3b0fcaf59a WatchSource:0}: Error finding container 7bb1d06ee5189f5bb1eaebe2571598d8ed0f2279c814aab840512a3b0fcaf59a: Status 404 returned error can't find the container with id 7bb1d06ee5189f5bb1eaebe2571598d8ed0f2279c814aab840512a3b0fcaf59a
Nov 02 15:04:31.767677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:31.767640    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg" event={"ID":"214f8315-f258-40d4-bb11-bfae94d43ff1","Type":"ContainerStarted","Data":"7bb1d06ee5189f5bb1eaebe2571598d8ed0f2279c814aab840512a3b0fcaf59a"}
Nov 02 15:04:32.773946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:32.773911    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg" event={"ID":"214f8315-f258-40d4-bb11-bfae94d43ff1","Type":"ContainerStarted","Data":"0a376dc77a370e27fbe43f0400be9dde08be943d60950988f78405a9ae8cbc1e"}
Nov 02 15:04:32.817059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:32.816996    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg" podStartSLOduration=1.609076636 podStartE2EDuration="1.816978737s" totalImagesPullingTime="207.902101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:32.813323119 +0000 UTC m=+525.071392569" watchObservedRunningTime="2026-11-02 15:04:32.816978737 +0000 UTC m=+525.075048191"
Nov 02 15:04:33.431677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:33.431634    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:36.767165 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:04:36.767117    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:04:37.942869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:37.942833    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:37.955440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:37.955417    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:38.816131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:38.816099    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89"
Nov 02 15:04:41.431358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:41.431327    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:41.444143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:41.444118    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:41.832412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:41.832316    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg"
Nov 02 15:04:45.483292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.483257    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-zq7s4"]
Nov 02 15:04:45.486094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.486076    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.500112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.500082    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 15:04:45.511676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.511644    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-z7mgl\""
Nov 02 15:04:45.514461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.514438    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-zq7s4"]
Nov 02 15:04:45.540638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.540610    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 15:04:45.608255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.608223    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs\") pod \"ai-gateway-controller-79567948bc-zq7s4\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") " pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.608426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.608267    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zg7\" (UniqueName: \"kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7\") pod \"ai-gateway-controller-79567948bc-zq7s4\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") " pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.709273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.709238    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs\") pod \"ai-gateway-controller-79567948bc-zq7s4\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") " pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.709543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.709286    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zg7\" (UniqueName: \"kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7\") pod \"ai-gateway-controller-79567948bc-zq7s4\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") " pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.711690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.711664    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs\") pod \"ai-gateway-controller-79567948bc-zq7s4\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") " pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.732460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.732436    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zg7\" (UniqueName: \"kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7\") pod \"ai-gateway-controller-79567948bc-zq7s4\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") " pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:45.799135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:45.799063    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:46.097528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:46.097496    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-zq7s4"]
Nov 02 15:04:46.099224 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:04:46.099187    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d00a999_d642_4b56_8a06_75e6a558b539.slice/crio-e0fae71a86a1d32019042b3136ba50491252fa13189509c60ff25e59220f1066 WatchSource:0}: Error finding container e0fae71a86a1d32019042b3136ba50491252fa13189509c60ff25e59220f1066: Status 404 returned error can't find the container with id e0fae71a86a1d32019042b3136ba50491252fa13189509c60ff25e59220f1066
Nov 02 15:04:46.852144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:46.852105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4" event={"ID":"2d00a999-d642-4b56-8a06-75e6a558b539","Type":"ContainerStarted","Data":"e0fae71a86a1d32019042b3136ba50491252fa13189509c60ff25e59220f1066"}
Nov 02 15:04:48.571486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:48.571460    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 15:04:48.864893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:48.864806    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4" event={"ID":"2d00a999-d642-4b56-8a06-75e6a558b539","Type":"ContainerStarted","Data":"7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd"}
Nov 02 15:04:48.865034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:48.864923    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:04:48.906163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:48.906112    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4" podStartSLOduration=1.449158062 podStartE2EDuration="3.906099779s" totalImagesPullingTime="2.456941717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:48.904768139 +0000 UTC m=+541.162837590" watchObservedRunningTime="2026-11-02 15:04:48.906099779 +0000 UTC m=+541.164169228"
Nov 02 15:04:50.811167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.811128    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"]
Nov 02 15:04:50.813892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.813872    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:50.837349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.837318    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"]
Nov 02 15:04:50.853246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.853215    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44bb49df-a7f6-426b-812b-df7db3361522-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s4wx8\" (UID: \"44bb49df-a7f6-426b-812b-df7db3361522\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:50.853517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.853495    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59r6\" (UniqueName: \"kubernetes.io/projected/44bb49df-a7f6-426b-812b-df7db3361522-kube-api-access-q59r6\") pod \"payload-pre-processing-7b98bb678c-s4wx8\" (UID: \"44bb49df-a7f6-426b-812b-df7db3361522\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:50.858258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.858236    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jzzvg\""
Nov 02 15:04:50.893768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.893731    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 15:04:50.954326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.954280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59r6\" (UniqueName: \"kubernetes.io/projected/44bb49df-a7f6-426b-812b-df7db3361522-kube-api-access-q59r6\") pod \"payload-pre-processing-7b98bb678c-s4wx8\" (UID: \"44bb49df-a7f6-426b-812b-df7db3361522\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:50.954534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.954392    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44bb49df-a7f6-426b-812b-df7db3361522-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s4wx8\" (UID: \"44bb49df-a7f6-426b-812b-df7db3361522\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:50.955129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.955108    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44bb49df-a7f6-426b-812b-df7db3361522-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s4wx8\" (UID: \"44bb49df-a7f6-426b-812b-df7db3361522\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:50.976817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:50.976786    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59r6\" (UniqueName: \"kubernetes.io/projected/44bb49df-a7f6-426b-812b-df7db3361522-kube-api-access-q59r6\") pod \"payload-pre-processing-7b98bb678c-s4wx8\" (UID: \"44bb49df-a7f6-426b-812b-df7db3361522\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:51.103013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.102939    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"]
Nov 02 15:04:51.105620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.105597    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"]
Nov 02 15:04:51.105871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.105847    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.108167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.108147    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.125869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.125847    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:51.131625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.131595    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"]
Nov 02 15:04:51.135497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.135472    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"]
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159410    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aac315-0378-4703-bd0b-67c163800196-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159468    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159504    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159533    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159578    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc22p\" (UniqueName: \"kubernetes.io/projected/67aac315-0378-4703-bd0b-67c163800196-kube-api-access-bc22p\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159638    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aac315-0378-4703-bd0b-67c163800196-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159750    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159776    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159815    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aac315-0378-4703-bd0b-67c163800196-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159869    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159904    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7qhq\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159935    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159961    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.159992    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.160026    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.163406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.160057    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.215314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.215286    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwrm4\""
Nov 02 15:04:51.260839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.260807    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.260847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7qhq\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.260866    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.260889    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.260911    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261040    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261086    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261158    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aac315-0378-4703-bd0b-67c163800196-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261190    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261219    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261307    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261391    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261445    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc22p\" (UniqueName: \"kubernetes.io/projected/67aac315-0378-4703-bd0b-67c163800196-kube-api-access-bc22p\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261503    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261553    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aac315-0378-4703-bd0b-67c163800196-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261580    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261615    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261640    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261648    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.261773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261678    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aac315-0378-4703-bd0b-67c163800196-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.262628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261819    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.262628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261906    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.262628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.261933    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.262628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.262202    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.262628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.262337    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.262628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.262435    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aac315-0378-4703-bd0b-67c163800196-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.263754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.263722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.263922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.263906    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aac315-0378-4703-bd0b-67c163800196-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.264456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.264440    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.264552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.264532    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aac315-0378-4703-bd0b-67c163800196-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.283738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.283714    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7qhq\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.283860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.283792    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc22p\" (UniqueName: \"kubernetes.io/projected/67aac315-0378-4703-bd0b-67c163800196-kube-api-access-bc22p\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.284590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.284572    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8q6sb\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.284704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.284686    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aac315-0378-4703-bd0b-67c163800196-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jx4lc\" (UID: \"67aac315-0378-4703-bd0b-67c163800196\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.419991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.419961    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:51.427898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.427865    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:51.430909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.430887    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"]
Nov 02 15:04:51.432741 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:04:51.432715    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bb49df_a7f6_426b_812b_df7db3361522.slice/crio-9e39cb91283e367d8ff3f70db369822415bfa79ac933a1b602aac92a939ec03b WatchSource:0}: Error finding container 9e39cb91283e367d8ff3f70db369822415bfa79ac933a1b602aac92a939ec03b: Status 404 returned error can't find the container with id 9e39cb91283e367d8ff3f70db369822415bfa79ac933a1b602aac92a939ec03b
Nov 02 15:04:51.739067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.739034    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"]
Nov 02 15:04:51.740210 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:04:51.740184    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8c355a_4867_4eee_98a4_0ccca21f6301.slice/crio-c2fd6ca4502596df8dfc513c4660e40cf30aac2bfe004c0c02aa8b2a5dea0bf4 WatchSource:0}: Error finding container c2fd6ca4502596df8dfc513c4660e40cf30aac2bfe004c0c02aa8b2a5dea0bf4: Status 404 returned error can't find the container with id c2fd6ca4502596df8dfc513c4660e40cf30aac2bfe004c0c02aa8b2a5dea0bf4
Nov 02 15:04:51.750959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.750789    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"]
Nov 02 15:04:51.753505 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:04:51.753482    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aac315_0378_4703_bd0b_67c163800196.slice/crio-5ee9af11133d78e33dd6d017e7050884b52c8856e722331c8e40a2f2cc572490 WatchSource:0}: Error finding container 5ee9af11133d78e33dd6d017e7050884b52c8856e722331c8e40a2f2cc572490: Status 404 returned error can't find the container with id 5ee9af11133d78e33dd6d017e7050884b52c8856e722331c8e40a2f2cc572490
Nov 02 15:04:51.890612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.890568    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8" event={"ID":"44bb49df-a7f6-426b-812b-df7db3361522","Type":"ContainerStarted","Data":"9e39cb91283e367d8ff3f70db369822415bfa79ac933a1b602aac92a939ec03b"}
Nov 02 15:04:51.892435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.892343    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" event={"ID":"67aac315-0378-4703-bd0b-67c163800196","Type":"ContainerStarted","Data":"5ee9af11133d78e33dd6d017e7050884b52c8856e722331c8e40a2f2cc572490"}
Nov 02 15:04:51.893997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:51.893936    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" event={"ID":"6c8c355a-4867-4eee-98a4-0ccca21f6301","Type":"ContainerStarted","Data":"c2fd6ca4502596df8dfc513c4660e40cf30aac2bfe004c0c02aa8b2a5dea0bf4"}
Nov 02 15:04:53.910503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:53.910464    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8" event={"ID":"44bb49df-a7f6-426b-812b-df7db3361522","Type":"ContainerStarted","Data":"18b6555f875f47fc51576df7c773fc4984a8a321468d86c9f052fb4439234a22"}
Nov 02 15:04:53.910953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:53.910549    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:04:53.954674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:53.954622    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8" podStartSLOduration=1.600045257 podStartE2EDuration="3.954605025s" totalImagesPullingTime="2.354559768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:53.950297617 +0000 UTC m=+546.208367070" watchObservedRunningTime="2026-11-02 15:04:53.954605025 +0000 UTC m=+546.212674476"
Nov 02 15:04:54.786491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.786460    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:04:54.786579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.786528    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:04:54.786579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.786555    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:04:54.885389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.885326    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:04:54.885511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.885428    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:04:54.885511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.885464    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:04:54.918844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.918810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" event={"ID":"6c8c355a-4867-4eee-98a4-0ccca21f6301","Type":"ContainerStarted","Data":"64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d"}
Nov 02 15:04:54.977283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:54.977223    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podStartSLOduration=0.933966732 podStartE2EDuration="3.97720174s" totalImagesPullingTime="3.043235008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:54.973384812 +0000 UTC m=+547.231454278" watchObservedRunningTime="2026-11-02 15:04:54.97720174 +0000 UTC m=+547.235271193"
Nov 02 15:04:55.424387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:55.420776    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:04:55.429393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:55.428499    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:04:55.429393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:55.428564    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:04:55.967190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:55.967144    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" event={"ID":"67aac315-0378-4703-bd0b-67c163800196","Type":"ContainerStarted","Data":"6c53e043dc44f5ce580b13e8e95221b892692bbc2479f5eed15dfb9889eccba7"}
Nov 02 15:04:56.043387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:56.042905    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podStartSLOduration=1.914024719 podStartE2EDuration="5.042886345s" totalImagesPullingTime="3.128861626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:56.040151595 +0000 UTC m=+548.298221046" watchObservedRunningTime="2026-11-02 15:04:56.042886345 +0000 UTC m=+548.300955797"
Nov 02 15:04:56.420855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:56.420823    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:04:56.421132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:56.421100    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:04:56.444406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:56.444356    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:04:56.444698 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:56.444674    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:04:56.444879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:56.444852    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:04:57.433389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:57.421821    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:04:57.433389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:57.421892    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:04:57.433389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:57.431334    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:04:57.433389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:57.431407    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:04:58.422423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:58.422385    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:04:58.422733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:58.422703    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:04:58.430718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:58.430687    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:04:58.430928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:58.430903    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:04:59.430498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:59.430460    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:04:59.431087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:59.431056    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:04:59.431487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:59.431459    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:04:59.431685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:59.431661    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:04:59.877234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:04:59.877143    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:05:00.420768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:00.420729    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:05:00.421068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:00.421031    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:05:00.444284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:00.444249    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:05:00.444868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:00.444837    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:05:01.421142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:01.421100    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:05:01.421293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:01.421179    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:05:01.428424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:01.428400    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:05:01.428537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:01.428442    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:05:02.421002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:02.420961    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8q6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:05:02.421411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:02.421043    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 15:05:02.428491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:02.428465    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:05:02.428562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:02.428516    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 15:05:03.424929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:03.424898    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:05:03.425287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:03.425170    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:05:03.425879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:03.425857    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:05:03.432357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:03.432338    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:05:03.432635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:03.432619    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:05:03.433177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:03.433163    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc"
Nov 02 15:05:04.921554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:04.921525    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s4wx8"
Nov 02 15:05:06.154855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.154816    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"]
Nov 02 15:05:06.158808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.158784    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.192261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.192226    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"]
Nov 02 15:05:06.264924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.264869    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.265500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.265473    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.265885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.265842    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.266126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.266108    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.266350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.266331    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbq7h\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.266566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.266549    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.266807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.266772    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.266912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.266881    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.267006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.266985    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.367990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368031    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbq7h\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368053    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368173    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368225    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368270    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368302    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368477    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368514    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368537    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368613    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368672    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.368832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.368753    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.369155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.369131    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.370517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.370499    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.370839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.370818    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.390870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.390843    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.390975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.390959    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbq7h\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jlsxv\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.473041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.473007    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:06.768013 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:05:06.767929    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:05:06.786927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.786899    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"]
Nov 02 15:05:06.788567 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:05:06.788536    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d8860_2269_40da_acb1_bccef5de32e9.slice/crio-dfed9fd56eb3fcd1348cafdba5ee4d2412d3c59f4578728af76428e4b5bd1ebb WatchSource:0}: Error finding container dfed9fd56eb3fcd1348cafdba5ee4d2412d3c59f4578728af76428e4b5bd1ebb: Status 404 returned error can't find the container with id dfed9fd56eb3fcd1348cafdba5ee4d2412d3c59f4578728af76428e4b5bd1ebb
Nov 02 15:05:06.791694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.791658    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:05:06.791789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.791726    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:05:06.791789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:06.791755    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:05:07.032430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:07.032330    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" event={"ID":"417d8860-2269-40da-acb1-bccef5de32e9","Type":"ContainerStarted","Data":"5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b"}
Nov 02 15:05:07.032430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:07.032403    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" event={"ID":"417d8860-2269-40da-acb1-bccef5de32e9","Type":"ContainerStarted","Data":"dfed9fd56eb3fcd1348cafdba5ee4d2412d3c59f4578728af76428e4b5bd1ebb"}
Nov 02 15:05:07.079013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:07.078945    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podStartSLOduration=1.078924191 podStartE2EDuration="1.078924191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:07.076403645 +0000 UTC m=+559.334473096" watchObservedRunningTime="2026-11-02 15:05:07.078924191 +0000 UTC m=+559.336993642"
Nov 02 15:05:07.473883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:07.473849    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:07.475494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:07.475467    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:07.475606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:07.475514    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:08.474157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:08.474118    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:08.474580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:08.474172    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:09.474095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:09.474055    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:09.474280 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:09.474113    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:10.473270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:10.473229    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:10.473469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:10.473289    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:11.473977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:11.473943    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:11.474399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:11.473999    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:12.474218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:12.474184    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:12.474626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:12.474237    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:13.473655 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:13.473619    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:05:13.473834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:13.473676    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 15:05:14.955002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:14.954965    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:14.955492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:14.955286    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:14.956206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:14.956178    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:05:15.002399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.002354    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-zq7s4"]
Nov 02 15:05:15.002570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.002549    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4" podUID="2d00a999-d642-4b56-8a06-75e6a558b539" containerName="manager" containerID="cri-o://7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd" gracePeriod=30
Nov 02 15:05:15.278893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.278865    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:05:15.375226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.375192    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs\") pod \"2d00a999-d642-4b56-8a06-75e6a558b539\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") "
Nov 02 15:05:15.375406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.375348    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7\" (UniqueName: \"kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7\") pod \"2d00a999-d642-4b56-8a06-75e6a558b539\" (UID: \"2d00a999-d642-4b56-8a06-75e6a558b539\") "
Nov 02 15:05:15.377233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.377207    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs" pod "2d00a999-d642-4b56-8a06-75e6a558b539" (UID: "2d00a999-d642-4b56-8a06-75e6a558b539"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:05:15.377321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.377226    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7" pod "2d00a999-d642-4b56-8a06-75e6a558b539" (UID: "2d00a999-d642-4b56-8a06-75e6a558b539"). InnerVolumeSpecName "kube-api-access-86zg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:05:15.476699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.476674    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zg7\" (UniqueName: \"kubernetes.io/projected/2d00a999-d642-4b56-8a06-75e6a558b539-kube-api-access-86zg7\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:05:15.476699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:15.476699    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d00a999-d642-4b56-8a06-75e6a558b539-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:05:16.091120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.091081    5041 generic.go:356] "Generic (PLEG): container finished" podID="2d00a999-d642-4b56-8a06-75e6a558b539" containerID="7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd" exitCode=0
Nov 02 15:05:16.091625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.091131    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4"
Nov 02 15:05:16.091625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.091163    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4" event={"ID":"2d00a999-d642-4b56-8a06-75e6a558b539","Type":"ContainerDied","Data":"7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd"}
Nov 02 15:05:16.091625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.091218    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-zq7s4" event={"ID":"2d00a999-d642-4b56-8a06-75e6a558b539","Type":"ContainerDied","Data":"e0fae71a86a1d32019042b3136ba50491252fa13189509c60ff25e59220f1066"}
Nov 02 15:05:16.091625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.091241    5041 scope.go:122] "RemoveContainer" containerID="7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd"
Nov 02 15:05:16.100598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.100580    5041 scope.go:122] "RemoveContainer" containerID="7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd"
Nov 02 15:05:16.100858 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:05:16.100837    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd\": container with ID starting with 7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd not found: ID does not exist" containerID="7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd"
Nov 02 15:05:16.100931 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.100864    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd"} err="failed to get container status \"7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd\": rpc error: code = NotFound desc = could not find container \"7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd\": container with ID starting with 7b4f367d09caf817d5df36f349cc592c66dd725718c3b43bbda61c274996c3cd not found: ID does not exist"
Nov 02 15:05:16.143772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.143702    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-zq7s4"]
Nov 02 15:05:16.154675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:16.154646    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-zq7s4"]
Nov 02 15:05:17.887521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:17.887489    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d00a999-d642-4b56-8a06-75e6a558b539" path="/var/lib/kubelet/pods/2d00a999-d642-4b56-8a06-75e6a558b539/volumes"
Nov 02 15:05:36.767262 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:05:36.767181    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:05:47.956427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:47.956399    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:05:47.957066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:47.957047    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:05:58.873563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.873525    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"]
Nov 02 15:05:58.874187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.874166    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2d00a999-d642-4b56-8a06-75e6a558b539" containerName="manager"
Nov 02 15:05:58.874247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.874191    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d00a999-d642-4b56-8a06-75e6a558b539" containerName="manager"
Nov 02 15:05:58.874322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.874311    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d00a999-d642-4b56-8a06-75e6a558b539" containerName="manager"
Nov 02 15:05:58.878881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.878864    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:58.904811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.904790    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-624254\""
Nov 02 15:05:58.904920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.904827    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-624254-dockercfg-9v6k2\""
Nov 02 15:05:58.910547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.910527    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"]
Nov 02 15:05:58.963147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.963089    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czlt\" (UniqueName: \"kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt\") pod \"payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:58.963899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.963855    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:58.964730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.964355    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"]
Nov 02 15:05:58.970303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.970283    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"]
Nov 02 15:05:58.970445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:58.970402    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.065167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.065128    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:59.065342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.065209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.065342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.065286    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6l6\" (UniqueName: \"kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6\") pod \"payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.065505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.065410    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czlt\" (UniqueName: \"kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt\") pod \"payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:59.066063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.066036    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:59.088673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.088652    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czlt\" (UniqueName: \"kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt\") pod \"payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:59.166050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.165986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6l6\" (UniqueName: \"kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6\") pod \"payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.166169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.166085    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.166765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.166743    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.188781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.188754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6l6\" (UniqueName: \"kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6\") pod \"payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.191469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.191448    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:05:59.293519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.293492    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:05:59.489528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.489502    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"]
Nov 02 15:05:59.491482 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:05:59.491457    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5421600_ded3_4025_9ea9_239c536e94cb.slice/crio-e96a66d92242743fd1eab3fe09268d0ed43418f4b531c0dc8ad54560ff00f81f WatchSource:0}: Error finding container e96a66d92242743fd1eab3fe09268d0ed43418f4b531c0dc8ad54560ff00f81f: Status 404 returned error can't find the container with id e96a66d92242743fd1eab3fe09268d0ed43418f4b531c0dc8ad54560ff00f81f
Nov 02 15:05:59.592137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:05:59.592113    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"]
Nov 02 15:05:59.593483 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:05:59.593461    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b7111b_0097_4bba_84b5_bf0493ee7910.slice/crio-8f92f59af63908ad25f2cd9fc3cccce89d8b3943e9376c52f2ad06aba6dccc42 WatchSource:0}: Error finding container 8f92f59af63908ad25f2cd9fc3cccce89d8b3943e9376c52f2ad06aba6dccc42: Status 404 returned error can't find the container with id 8f92f59af63908ad25f2cd9fc3cccce89d8b3943e9376c52f2ad06aba6dccc42
Nov 02 15:06:00.054080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.054043    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"]
Nov 02 15:06:00.057959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.057939    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.075282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.075256    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-1cc1b2-serving-cert\""
Nov 02 15:06:00.089269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.088602    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cz4f2\""
Nov 02 15:06:00.089269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.088708    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:06:00.089269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.088691    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 15:06:00.089269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.088804    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:06:00.095593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.095566    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"]
Nov 02 15:06:00.176018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.175991    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.176166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.176033    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99j2\" (UniqueName: \"kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.176166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.176144    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.176255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.176178    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.277625    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.277719    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.277809    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.277841    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99j2\" (UniqueName: \"kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.278911    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.288754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.290399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.289259    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.319258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.318119    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99j2\" (UniqueName: \"kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2\") pod \"maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.348394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.348332    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" event={"ID":"c7b7111b-0097-4bba-84b5-bf0493ee7910","Type":"ContainerStarted","Data":"e26bf3e04032e52dba585ba0e6b760f1d52eb9b29cfd622ec359a1db61c0ad34"}
Nov 02 15:06:00.348566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.348448    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" event={"ID":"c7b7111b-0097-4bba-84b5-bf0493ee7910","Type":"ContainerStarted","Data":"8f92f59af63908ad25f2cd9fc3cccce89d8b3943e9376c52f2ad06aba6dccc42"}
Nov 02 15:06:00.348566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.348524    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:06:00.350634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.350606    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" event={"ID":"e5421600-ded3-4025-9ea9-239c536e94cb","Type":"ContainerStarted","Data":"03f5c9f252422a52b5a85d0929eb3aaf2ca44e3ce040a59d7bd46845403d8022"}
Nov 02 15:06:00.350750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.350657    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" event={"ID":"e5421600-ded3-4025-9ea9-239c536e94cb","Type":"ContainerStarted","Data":"e96a66d92242743fd1eab3fe09268d0ed43418f4b531c0dc8ad54560ff00f81f"}
Nov 02 15:06:00.350835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.350820    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:06:00.378031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.377806    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:00.394618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.394565    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" podStartSLOduration=2.394549348 podStartE2EDuration="2.394549348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:00.392431323 +0000 UTC m=+612.650500774" watchObservedRunningTime="2026-11-02 15:06:00.394549348 +0000 UTC m=+612.652618799"
Nov 02 15:06:00.440497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.439048    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" podStartSLOduration=2.439029602 podStartE2EDuration="2.439029602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:00.436285639 +0000 UTC m=+612.694355100" watchObservedRunningTime="2026-11-02 15:06:00.439029602 +0000 UTC m=+612.697099052"
Nov 02 15:06:00.545668 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.545634    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"]
Nov 02 15:06:00.550831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.550810    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.565146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.565120    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-1cc1b2\""
Nov 02 15:06:00.565270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.565257    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-1cc1b2-dockercfg-lqkpb\""
Nov 02 15:06:00.571420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.571355    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"]
Nov 02 15:06:00.655072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.655031    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"]
Nov 02 15:06:00.659109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.659088    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.680663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.680642    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.680792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.680708    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbp5\" (UniqueName: \"kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5\") pod \"payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.727549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.727528    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"]
Nov 02 15:06:00.729349 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:00.729325    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8acb4a_4891_4ae5_a8c6_f30721f52a08.slice/crio-4e7546ff5d1d5e573168f27e17a83c6f424b01d0ea4751e16a397fb07e750dfb WatchSource:0}: Error finding container 4e7546ff5d1d5e573168f27e17a83c6f424b01d0ea4751e16a397fb07e750dfb: Status 404 returned error can't find the container with id 4e7546ff5d1d5e573168f27e17a83c6f424b01d0ea4751e16a397fb07e750dfb
Nov 02 15:06:00.730106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.730083    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"]
Nov 02 15:06:00.781570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.781542    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.781653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.781581    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64bc\" (UniqueName: \"kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc\") pod \"payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.781653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.781639    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbp5\" (UniqueName: \"kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5\") pod \"payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.781780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.781755    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.782178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.782159    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.809229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.809198    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbp5\" (UniqueName: \"kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5\") pod \"payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.863844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.863790    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:00.882474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.882447    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.882584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.882496    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64bc\" (UniqueName: \"kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc\") pod \"payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.883253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.883229    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.906738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.906710    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64bc\" (UniqueName: \"kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc\") pod \"payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:00.972701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:00.972668    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:01.162271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.162235    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"]
Nov 02 15:06:01.163707 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:01.163661    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69736d7_8358_4118_ae72_c3517fc30b94.slice/crio-5340983e3b98854b5daeecf5f7d9f746dbc87cbc54870a091e0dcb639ada337e WatchSource:0}: Error finding container 5340983e3b98854b5daeecf5f7d9f746dbc87cbc54870a091e0dcb639ada337e: Status 404 returned error can't find the container with id 5340983e3b98854b5daeecf5f7d9f746dbc87cbc54870a091e0dcb639ada337e
Nov 02 15:06:01.274474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.274444    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"]
Nov 02 15:06:01.275680 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:01.275651    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c52bec_5157_4483_9df4_413a53514093.slice/crio-44a801a11a5d8879251c004289178b121cae767e17b1411ca15d5450c488aa9d WatchSource:0}: Error finding container 44a801a11a5d8879251c004289178b121cae767e17b1411ca15d5450c488aa9d: Status 404 returned error can't find the container with id 44a801a11a5d8879251c004289178b121cae767e17b1411ca15d5450c488aa9d
Nov 02 15:06:01.358761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.358733    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng" event={"ID":"ef8acb4a-4891-4ae5-a8c6-f30721f52a08","Type":"ContainerStarted","Data":"4e7546ff5d1d5e573168f27e17a83c6f424b01d0ea4751e16a397fb07e750dfb"}
Nov 02 15:06:01.360471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.360439    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" event={"ID":"c9c52bec-5157-4483-9df4-413a53514093","Type":"ContainerStarted","Data":"23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6"}
Nov 02 15:06:01.360592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.360500    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:01.360592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.360526    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" event={"ID":"c9c52bec-5157-4483-9df4-413a53514093","Type":"ContainerStarted","Data":"44a801a11a5d8879251c004289178b121cae767e17b1411ca15d5450c488aa9d"}
Nov 02 15:06:01.362172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.362144    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" event={"ID":"f69736d7-8358-4118-ae72-c3517fc30b94","Type":"ContainerStarted","Data":"60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458"}
Nov 02 15:06:01.362291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.362199    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" event={"ID":"f69736d7-8358-4118-ae72-c3517fc30b94","Type":"ContainerStarted","Data":"5340983e3b98854b5daeecf5f7d9f746dbc87cbc54870a091e0dcb639ada337e"}
Nov 02 15:06:01.362350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.362299    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:01.406371 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.406328    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" podStartSLOduration=1.4063154660000001 podStartE2EDuration="1.406315466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:01.40492981 +0000 UTC m=+613.662999263" watchObservedRunningTime="2026-11-02 15:06:01.406315466 +0000 UTC m=+613.664384960"
Nov 02 15:06:01.412113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.412057    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"]
Nov 02 15:06:01.416441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.416417    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.443596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.443530    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"]
Nov 02 15:06:01.444797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.444761    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5dd8e9\""
Nov 02 15:06:01.456518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.452018    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" podStartSLOduration=1.452003033 podStartE2EDuration="1.452003033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:01.449035559 +0000 UTC m=+613.707105012" watchObservedRunningTime="2026-11-02 15:06:01.452003033 +0000 UTC m=+613.710072483"
Nov 02 15:06:01.467753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.467732    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"]
Nov 02 15:06:01.471817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.471796    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.487908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.487873    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5dd8e9-dockercfg-k9jw8\""
Nov 02 15:06:01.514959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.514163    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"]
Nov 02 15:06:01.590586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.590535    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66rd\" (UniqueName: \"kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd\") pod \"payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.590768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.590717    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.590873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.590842    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.590993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.590885    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hl4m\" (UniqueName: \"kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m\") pod \"payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.692102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.692069    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66rd\" (UniqueName: \"kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd\") pod \"payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.692285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.692236    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.692351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.692299    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.692351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.692326    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hl4m\" (UniqueName: \"kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m\") pod \"payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.693060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.693037    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.693104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.693083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.715352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.715326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66rd\" (UniqueName: \"kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd\") pod \"payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.715493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.715326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hl4m\" (UniqueName: \"kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m\") pod \"payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:01.731834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.731809    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:01.789846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:01.789794    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:02.040216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.040187    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"]
Nov 02 15:06:02.041648 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:02.041612    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87376fee_2fff_4e42_af60_c78478971f37.slice/crio-aecd7b2f6f109a5e1f3922e3ad375d77df7e90578b07129f17697f1ba63e466d WatchSource:0}: Error finding container aecd7b2f6f109a5e1f3922e3ad375d77df7e90578b07129f17697f1ba63e466d: Status 404 returned error can't find the container with id aecd7b2f6f109a5e1f3922e3ad375d77df7e90578b07129f17697f1ba63e466d
Nov 02 15:06:02.129245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.129182    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"]
Nov 02 15:06:02.132030 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:02.131973    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2b628b_5a9a_4990_bd39_26bd0a01cca0.slice/crio-d1679b26025b6dde3532702fdd89b57e206ef2068ee0b29af5af35bfcbd79344 WatchSource:0}: Error finding container d1679b26025b6dde3532702fdd89b57e206ef2068ee0b29af5af35bfcbd79344: Status 404 returned error can't find the container with id d1679b26025b6dde3532702fdd89b57e206ef2068ee0b29af5af35bfcbd79344
Nov 02 15:06:02.371463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.371378    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" event={"ID":"87376fee-2fff-4e42-af60-c78478971f37","Type":"ContainerStarted","Data":"dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113"}
Nov 02 15:06:02.371463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.371431    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" event={"ID":"87376fee-2fff-4e42-af60-c78478971f37","Type":"ContainerStarted","Data":"aecd7b2f6f109a5e1f3922e3ad375d77df7e90578b07129f17697f1ba63e466d"}
Nov 02 15:06:02.372213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.372189    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:02.374062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.374036    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" event={"ID":"9c2b628b-5a9a-4990-bd39-26bd0a01cca0","Type":"ContainerStarted","Data":"c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3"}
Nov 02 15:06:02.374150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.374081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" event={"ID":"9c2b628b-5a9a-4990-bd39-26bd0a01cca0","Type":"ContainerStarted","Data":"d1679b26025b6dde3532702fdd89b57e206ef2068ee0b29af5af35bfcbd79344"}
Nov 02 15:06:02.413557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.413503    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" podStartSLOduration=1.413487473 podStartE2EDuration="1.413487473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:02.411150466 +0000 UTC m=+614.669219918" watchObservedRunningTime="2026-11-02 15:06:02.413487473 +0000 UTC m=+614.671556923"
Nov 02 15:06:02.455930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:02.455888    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" podStartSLOduration=1.455875199 podStartE2EDuration="1.455875199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:02.453575127 +0000 UTC m=+614.711644577" watchObservedRunningTime="2026-11-02 15:06:02.455875199 +0000 UTC m=+614.713944649"
Nov 02 15:06:03.152499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.152460    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"]
Nov 02 15:06:03.156644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.156621    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.170980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.170955    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6af50f\""
Nov 02 15:06:03.171151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.171132    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6af50f-dockercfg-847ls\""
Nov 02 15:06:03.179104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.179084    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"]
Nov 02 15:06:03.256591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.256550    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"]
Nov 02 15:06:03.262924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.262891    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.283450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.283415    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"]
Nov 02 15:06:03.314573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.314539    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4svx\" (UniqueName: \"kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx\") pod \"payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.314748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.314606    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.381442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.381405    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng" event={"ID":"ef8acb4a-4891-4ae5-a8c6-f30721f52a08","Type":"ContainerStarted","Data":"8763d2e978b9983277b64bf385cf999a8e8a9232d607703208e4e9dda767ef90"}
Nov 02 15:06:03.381904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.381886    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:03.381989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.381932    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:03.415203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.415129    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4svx\" (UniqueName: \"kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx\") pod \"payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.415203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.415181    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.415422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.415269    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.415422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.415295    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknm4\" (UniqueName: \"kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4\") pod \"payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.416139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.416113    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.424223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.423987    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng" podStartSLOduration=1.674576748 podStartE2EDuration="3.423972282s" totalImagesPullingTime="1.749395534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:03.421327372 +0000 UTC m=+615.679396822" watchObservedRunningTime="2026-11-02 15:06:03.423972282 +0000 UTC m=+615.682041735"
Nov 02 15:06:03.438441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.438421    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4svx\" (UniqueName: \"kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx\") pod \"payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.471227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.471204    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:03.517391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.516871    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.517391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.516942    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknm4\" (UniqueName: \"kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4\") pod \"payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.518151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.518114    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.541651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.541477    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknm4\" (UniqueName: \"kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4\") pod \"payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.582716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.582681    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:03.777979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.777947    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"]
Nov 02 15:06:03.779792 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:03.779765    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83af6371_cf0d_4d86_9b31_16e17a05e1d2.slice/crio-097df980f1a436003f5b8b560df478b053374a82d1d96e8b25f33e9f2b343b5a WatchSource:0}: Error finding container 097df980f1a436003f5b8b560df478b053374a82d1d96e8b25f33e9f2b343b5a: Status 404 returned error can't find the container with id 097df980f1a436003f5b8b560df478b053374a82d1d96e8b25f33e9f2b343b5a
Nov 02 15:06:03.891728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.891697    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"]
Nov 02 15:06:03.892169 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:03.892145    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50362a8d_c5af_4496_b343_b60ec54f3f0f.slice/crio-ee0d7df3d32bccf21ceff000d5faf522446cac4fc7c572faafe6c238095ed7a7 WatchSource:0}: Error finding container ee0d7df3d32bccf21ceff000d5faf522446cac4fc7c572faafe6c238095ed7a7: Status 404 returned error can't find the container with id ee0d7df3d32bccf21ceff000d5faf522446cac4fc7c572faafe6c238095ed7a7
Nov 02 15:06:03.952313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.952286    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"]
Nov 02 15:06:03.956227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.956211    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:03.970379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.970344    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-739bf1\""
Nov 02 15:06:03.970472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.970421    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-739bf1-dockercfg-x8h72\""
Nov 02 15:06:03.980339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:03.980310    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"]
Nov 02 15:06:04.039564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.039476    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnwr\" (UniqueName: \"kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr\") pod \"payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.039718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.039575    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.049563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.049534    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"]
Nov 02 15:06:04.057457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.057432    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.074752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.074725    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"]
Nov 02 15:06:04.141035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.141007    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr\") pod \"payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.141176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.141068    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnwr\" (UniqueName: \"kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr\") pod \"payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.141176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.141116    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.141296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.141254    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.142131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.142100    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.163500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.163474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnwr\" (UniqueName: \"kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr\") pod \"payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.247560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.243545    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.247560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.243759    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr\") pod \"payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.252114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.252091    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.269193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.269171    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr\") pod \"payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.269305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.269291    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:04.375916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.375876    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:04.395188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.395149    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" event={"ID":"50362a8d-c5af-4496-b343-b60ec54f3f0f","Type":"ContainerStarted","Data":"64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1"}
Nov 02 15:06:04.397745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.397714    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" event={"ID":"50362a8d-c5af-4496-b343-b60ec54f3f0f","Type":"ContainerStarted","Data":"ee0d7df3d32bccf21ceff000d5faf522446cac4fc7c572faafe6c238095ed7a7"}
Nov 02 15:06:04.398692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.398670    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:04.401139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.401106    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" event={"ID":"83af6371-cf0d-4d86-9b31-16e17a05e1d2","Type":"ContainerStarted","Data":"7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55"}
Nov 02 15:06:04.401273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.401163    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" event={"ID":"83af6371-cf0d-4d86-9b31-16e17a05e1d2","Type":"ContainerStarted","Data":"097df980f1a436003f5b8b560df478b053374a82d1d96e8b25f33e9f2b343b5a"}
Nov 02 15:06:04.401336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.401314    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:04.445951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.445878    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" podStartSLOduration=1.44586264 podStartE2EDuration="1.44586264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:04.439937513 +0000 UTC m=+616.698006962" watchObservedRunningTime="2026-11-02 15:06:04.44586264 +0000 UTC m=+616.703932090"
Nov 02 15:06:04.480393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.480308    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" podStartSLOduration=1.48029471 podStartE2EDuration="1.48029471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:04.479936061 +0000 UTC m=+616.738005514" watchObservedRunningTime="2026-11-02 15:06:04.48029471 +0000 UTC m=+616.738364162"
Nov 02 15:06:04.567119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.567092    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"]
Nov 02 15:06:04.568226 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:04.568169    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa3eea8_a2a6_4608_b93e_dfce5eec5034.slice/crio-34957b3de1fbe0d03a0adb548b923f6ba3f52801ee9c468bbf7265075c063d20 WatchSource:0}: Error finding container 34957b3de1fbe0d03a0adb548b923f6ba3f52801ee9c468bbf7265075c063d20: Status 404 returned error can't find the container with id 34957b3de1fbe0d03a0adb548b923f6ba3f52801ee9c468bbf7265075c063d20
Nov 02 15:06:04.709144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:04.709122    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"]
Nov 02 15:06:04.711268 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:04.711245    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4307bc_148b_44b3_a773_a137aad7ca1b.slice/crio-99a8a2b912231e0bf1d13ff7081a3184c81c8211ea6975600aab46dd407e774d WatchSource:0}: Error finding container 99a8a2b912231e0bf1d13ff7081a3184c81c8211ea6975600aab46dd407e774d: Status 404 returned error can't find the container with id 99a8a2b912231e0bf1d13ff7081a3184c81c8211ea6975600aab46dd407e774d
Nov 02 15:06:05.409902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.409867    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" event={"ID":"2aa3eea8-a2a6-4608-b93e-dfce5eec5034","Type":"ContainerStarted","Data":"61fc4f54ee902185613e82743d47b91f04f0ceaef181c7384fb647f615bd3e78"}
Nov 02 15:06:05.410404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.409928    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:05.410404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.409951    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" event={"ID":"2aa3eea8-a2a6-4608-b93e-dfce5eec5034","Type":"ContainerStarted","Data":"34957b3de1fbe0d03a0adb548b923f6ba3f52801ee9c468bbf7265075c063d20"}
Nov 02 15:06:05.411948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.411910    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" event={"ID":"de4307bc-148b-44b3-a773-a137aad7ca1b","Type":"ContainerStarted","Data":"5239776d46447a2ef17e2dc061555d87046f4b4abee3b14c868a0bb6c00a4277"}
Nov 02 15:06:05.412054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.411956    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" event={"ID":"de4307bc-148b-44b3-a773-a137aad7ca1b","Type":"ContainerStarted","Data":"99a8a2b912231e0bf1d13ff7081a3184c81c8211ea6975600aab46dd407e774d"}
Nov 02 15:06:05.412054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.411976    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:05.449525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.449473    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" podStartSLOduration=2.449459021 podStartE2EDuration="2.449459021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:05.448264053 +0000 UTC m=+617.706333503" watchObservedRunningTime="2026-11-02 15:06:05.449459021 +0000 UTC m=+617.707528471"
Nov 02 15:06:05.495807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:05.495756    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" podStartSLOduration=1.495739883 podStartE2EDuration="1.495739883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:05.491583194 +0000 UTC m=+617.749652667" watchObservedRunningTime="2026-11-02 15:06:05.495739883 +0000 UTC m=+617.753809334"
Nov 02 15:06:06.770760 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:06:06.770686    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:06:08.743200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.743168    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"]
Nov 02 15:06:08.748267 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.748249    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:08.772689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.772667    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5113c2\""
Nov 02 15:06:08.784174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.784149    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5113c2-dockercfg-6dhmg\""
Nov 02 15:06:08.786837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.786814    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"]
Nov 02 15:06:08.894386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.893510    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62ch\" (UniqueName: \"kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch\") pod \"payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:08.894386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.893592    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:08.895416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.895382    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"]
Nov 02 15:06:08.900163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.900141    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:08.938803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.938768    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"]
Nov 02 15:06:08.994425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.994334    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62ch\" (UniqueName: \"kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch\") pod \"payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:08.994570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.994448    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:08.994570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.994486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:08.994570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.994559    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx2h\" (UniqueName: \"kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h\") pod \"payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:08.995161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:08.995141    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:09.018068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.018029    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62ch\" (UniqueName: \"kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch\") pod \"payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:09.062188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.062155    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:09.096030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.095988    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:09.096170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.096112    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx2h\" (UniqueName: \"kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h\") pod \"payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:09.096710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.096689    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:09.119628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.119605    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx2h\" (UniqueName: \"kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h\") pod \"payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:09.214698 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.214666    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:09.376536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.376505    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"]
Nov 02 15:06:09.376735 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:09.376712    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7446d1_5270_4146_af20_4dedf8f877a8.slice/crio-1d29435784b3a8b5d697982e90cbcb130820748ddd3111cac6343701ffd096e9 WatchSource:0}: Error finding container 1d29435784b3a8b5d697982e90cbcb130820748ddd3111cac6343701ffd096e9: Status 404 returned error can't find the container with id 1d29435784b3a8b5d697982e90cbcb130820748ddd3111cac6343701ffd096e9
Nov 02 15:06:09.407267 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.407246    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:06:09.436054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.436031    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" event={"ID":"dc7446d1-5270-4146-af20-4dedf8f877a8","Type":"ContainerStarted","Data":"1d29435784b3a8b5d697982e90cbcb130820748ddd3111cac6343701ffd096e9"}
Nov 02 15:06:09.518439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.518409    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"]
Nov 02 15:06:09.521732 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:09.521707    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44955817_1b51_4bbf_a3f9_3c3fbb2a16c2.slice/crio-5d733165e1319ef10439ac5d3e20159d7f127c02e16b227b3721b6fe45277a3f WatchSource:0}: Error finding container 5d733165e1319ef10439ac5d3e20159d7f127c02e16b227b3721b6fe45277a3f: Status 404 returned error can't find the container with id 5d733165e1319ef10439ac5d3e20159d7f127c02e16b227b3721b6fe45277a3f
Nov 02 15:06:09.840404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.840302    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"]
Nov 02 15:06:09.844500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.844478    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:09.866332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.866300    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7ecfe1\""
Nov 02 15:06:09.866332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.866303    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7ecfe1-dockercfg-lhw7s\""
Nov 02 15:06:09.869489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:09.869462    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"]
Nov 02 15:06:10.004463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.004419    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knjd\" (UniqueName: \"kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd\") pod \"payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.004651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.004553    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.105689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.105607    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knjd\" (UniqueName: \"kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd\") pod \"payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.105689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.105683    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.106413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.106394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.128894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.128863    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knjd\" (UniqueName: \"kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd\") pod \"payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.158546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.158517    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:10.448186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.448148    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" event={"ID":"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2","Type":"ContainerStarted","Data":"e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96"}
Nov 02 15:06:10.448400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.448211    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" event={"ID":"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2","Type":"ContainerStarted","Data":"5d733165e1319ef10439ac5d3e20159d7f127c02e16b227b3721b6fe45277a3f"}
Nov 02 15:06:10.448400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.448239    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:10.449718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.449694    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" event={"ID":"dc7446d1-5270-4146-af20-4dedf8f877a8","Type":"ContainerStarted","Data":"c093bb1b5a7ec72e223280bef7681b869c8dafd9ede251c25811bc122ca3408b"}
Nov 02 15:06:10.449847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.449827    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:10.458942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.458920    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"]
Nov 02 15:06:10.460992 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:10.460972    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870134d6_03de_4a45_b54f_9576327ac3d9.slice/crio-95516ccbb5d6d9a2982d178c98fc2e32342bd1ca40219740b43ffcf4fe90a0e1 WatchSource:0}: Error finding container 95516ccbb5d6d9a2982d178c98fc2e32342bd1ca40219740b43ffcf4fe90a0e1: Status 404 returned error can't find the container with id 95516ccbb5d6d9a2982d178c98fc2e32342bd1ca40219740b43ffcf4fe90a0e1
Nov 02 15:06:10.494846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.492473    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" podStartSLOduration=2.492454192 podStartE2EDuration="2.492454192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:10.489547442 +0000 UTC m=+622.747616893" watchObservedRunningTime="2026-11-02 15:06:10.492454192 +0000 UTC m=+622.750523644"
Nov 02 15:06:10.531866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:10.531812    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" podStartSLOduration=2.531798763 podStartE2EDuration="2.531798763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:10.529571134 +0000 UTC m=+622.787640584" watchObservedRunningTime="2026-11-02 15:06:10.531798763 +0000 UTC m=+622.789868213"
Nov 02 15:06:11.372971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:11.372936    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:06:11.373509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:11.373376    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:06:11.457821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:11.457786    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" event={"ID":"870134d6-03de-4a45-b54f-9576327ac3d9","Type":"ContainerStarted","Data":"857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07"}
Nov 02 15:06:11.458003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:11.457843    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:11.458003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:11.457858    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" event={"ID":"870134d6-03de-4a45-b54f-9576327ac3d9","Type":"ContainerStarted","Data":"95516ccbb5d6d9a2982d178c98fc2e32342bd1ca40219740b43ffcf4fe90a0e1"}
Nov 02 15:06:11.508060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:11.507999    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" podStartSLOduration=2.507979401 podStartE2EDuration="2.507979401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:11.506010209 +0000 UTC m=+623.764079673" watchObservedRunningTime="2026-11-02 15:06:11.507979401 +0000 UTC m=+623.766048852"
Nov 02 15:06:12.377172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:12.377133    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:06:12.377651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:12.377219    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:06:14.405032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:14.405001    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:06:14.405462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:14.405086    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:06:15.414819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:15.414787    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:06:16.420835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:16.420802    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:06:16.422000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:16.421976    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:06:16.422137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:16.422055    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:06:21.461558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:21.461527    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:06:21.462057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:21.461610    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:06:22.467612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:22.467581    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:06:36.768381 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:06:36.768323    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:06:50.762675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.762628    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"]
Nov 02 15:06:50.766942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.766918    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.792708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.792683    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"simulator-w6-5113c2-kserve-self-signed-certs\""
Nov 02 15:06:50.792885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.792711    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"openshift-service-ca.crt\""
Nov 02 15:06:50.792885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.792725    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"default-dockercfg-dn9b4\""
Nov 02 15:06:50.793016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.792939    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"kube-root-ca.crt\""
Nov 02 15:06:50.795824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.795800    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"]
Nov 02 15:06:50.856799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.856774    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f982z\" (UniqueName: \"kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.856946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.856821    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.856946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.856853    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.857038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.856961    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.857084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.857037    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.857128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.857076    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.957870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.957845    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.957991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.957878    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.957991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.957897    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.957991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.957963    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f982z\" (UniqueName: \"kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.957991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.957988    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.958208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.958004    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.958356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.958337    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.958457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.958431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.958499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.958456    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.960072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.960051    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.960278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.960260    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:50.980966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:50.980943    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f982z\" (UniqueName: \"kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z\") pod \"simulator-w6-5113c2-kserve-65b959864-qspcc\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") " pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:51.079702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:51.079617    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:06:51.379635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:51.379611    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"]
Nov 02 15:06:51.381069 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:06:51.381044    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fbee385_a651_4bc8_82cd_c35a08875293.slice/crio-56eb2917a816dd9bd8ae3eef380f3d02d4bafa4aa1506917286196017ebbe417 WatchSource:0}: Error finding container 56eb2917a816dd9bd8ae3eef380f3d02d4bafa4aa1506917286196017ebbe417: Status 404 returned error can't find the container with id 56eb2917a816dd9bd8ae3eef380f3d02d4bafa4aa1506917286196017ebbe417
Nov 02 15:06:51.729212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:51.729175    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" event={"ID":"3fbee385-a651-4bc8-82cd-c35a08875293","Type":"ContainerStarted","Data":"efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0"}
Nov 02 15:06:51.729423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:51.729220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" event={"ID":"3fbee385-a651-4bc8-82cd-c35a08875293","Type":"ContainerStarted","Data":"56eb2917a816dd9bd8ae3eef380f3d02d4bafa4aa1506917286196017ebbe417"}
Nov 02 15:06:51.771633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:51.771588    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podStartSLOduration=1.771574891 podStartE2EDuration="1.771574891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:51.7712449 +0000 UTC m=+664.029314349" watchObservedRunningTime="2026-11-02 15:06:51.771574891 +0000 UTC m=+664.029644341"
Nov 02 15:06:52.079965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:06:52.079893    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:07:01.080064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:01.079988    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:07:01.092338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:01.092316    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:07:01.800726 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:01.800691    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:07:06.406044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.406009    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"]
Nov 02 15:07:06.409281 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.409263    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.424174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.424143    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"openshift-service-ca.crt\""
Nov 02 15:07:06.424279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.424215    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"default-dockercfg-4m4cf\""
Nov 02 15:07:06.436049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.436024    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"premium-w3-624254-kserve-self-signed-certs\""
Nov 02 15:07:06.441650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.441622    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"]
Nov 02 15:07:06.455429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.455401    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"kube-root-ca.crt\""
Nov 02 15:07:06.523169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.523134    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.523384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.523260    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.523384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.523309    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.523483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.523390    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.523483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.523435    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxqq\" (UniqueName: \"kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.523483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.523455    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625007    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625094    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625113    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625181    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxqq\" (UniqueName: \"kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625224    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625641    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625730    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.625878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.625830    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.627536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.627518    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.627879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.627858    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.649211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.649179    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxqq\" (UniqueName: \"kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq\") pod \"premium-w3-624254-kserve-6dbfd45d86-79zxp\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") " pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.724397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:06.724340    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:06.771809 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:07:06.771762    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:07:07.031962 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:07:07.031923    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a83c0e2_0bc1_4e1b_aead_6dab74b2e66f.slice/crio-ada6699ec2836dc2c234870c7cae6d8504b1260dd955676dec37136c20c038f7 WatchSource:0}: Error finding container ada6699ec2836dc2c234870c7cae6d8504b1260dd955676dec37136c20c038f7: Status 404 returned error can't find the container with id ada6699ec2836dc2c234870c7cae6d8504b1260dd955676dec37136c20c038f7
Nov 02 15:07:07.032225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:07.032205    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"]
Nov 02 15:07:07.849564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:07.849530    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" event={"ID":"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f","Type":"ContainerStarted","Data":"3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa"}
Nov 02 15:07:07.849564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:07.849570    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" event={"ID":"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f","Type":"ContainerStarted","Data":"ada6699ec2836dc2c234870c7cae6d8504b1260dd955676dec37136c20c038f7"}
Nov 02 15:07:07.891533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:07.891476    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podStartSLOduration=1.891458361 podStartE2EDuration="1.891458361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:07.889612055 +0000 UTC m=+680.147681505" watchObservedRunningTime="2026-11-02 15:07:07.891458361 +0000 UTC m=+680.149527826"
Nov 02 15:07:08.724864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:08.724830    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:16.724693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:16.724646    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:16.740712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:16.740688    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:16.917747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:16.917720    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:07:26.710608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.710574    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"]
Nov 02 15:07:26.714273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.714248    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.729714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.729392    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"openshift-service-ca.crt\""
Nov 02 15:07:26.740724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.740698    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"kube-root-ca.crt\""
Nov 02 15:07:26.740852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.740744    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"default-dockercfg-q4dms\""
Nov 02 15:07:26.740852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.740781    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"premium-w2-739bf1-kserve-self-signed-certs\""
Nov 02 15:07:26.745741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.745710    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"]
Nov 02 15:07:26.836701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.836668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.836852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.836742    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.836852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.836777    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.836943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.836878    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6t6\" (UniqueName: \"kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.836988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.836946    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.837025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.837006    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938310    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6t6\" (UniqueName: \"kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938387    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938455    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938491    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938538    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938806    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.938888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.938870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.939025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.939000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.940635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.940616    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.941024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.941005    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:26.960869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:26.960812    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6t6\" (UniqueName: \"kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6\") pod \"premium-w2-739bf1-kserve-6c597bbf56-lcrvp\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") " pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:27.027419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:27.027399    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:27.326206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:27.326178    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"]
Nov 02 15:07:27.327631 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:07:27.327603    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fff81be_b147_4d7c_b61f_0da2100e237b.slice/crio-b0a2d78077ec1d941df6810d7ee7115e96ea0cce7d380217b380cc929be50e83 WatchSource:0}: Error finding container b0a2d78077ec1d941df6810d7ee7115e96ea0cce7d380217b380cc929be50e83: Status 404 returned error can't find the container with id b0a2d78077ec1d941df6810d7ee7115e96ea0cce7d380217b380cc929be50e83
Nov 02 15:07:27.970919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:27.970889    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" event={"ID":"9fff81be-b147-4d7c-b61f-0da2100e237b","Type":"ContainerStarted","Data":"a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37"}
Nov 02 15:07:27.971279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:27.970930    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" event={"ID":"9fff81be-b147-4d7c-b61f-0da2100e237b","Type":"ContainerStarted","Data":"b0a2d78077ec1d941df6810d7ee7115e96ea0cce7d380217b380cc929be50e83"}
Nov 02 15:07:28.012322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:28.012280    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podStartSLOduration=2.012267535 podStartE2EDuration="2.012267535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:28.010919869 +0000 UTC m=+700.268989329" watchObservedRunningTime="2026-11-02 15:07:28.012267535 +0000 UTC m=+700.270336985"
Nov 02 15:07:28.028026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:28.028001    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:36.768919 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:07:36.768877    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:07:37.028468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:37.028392    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:37.041584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:37.041553    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:38.039271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:38.039240    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:07:58.119935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.119901    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"]
Nov 02 15:07:58.123756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.123728    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.137692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.137662    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:07:58.148884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.148859    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:07:58.153728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.153702    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"]
Nov 02 15:07:58.259008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.258976    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.259136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.259091    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.259182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.259172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwq6\" (UniqueName: \"kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.360063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.360026    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwq6\" (UniqueName: \"kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.360277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.360103    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.360277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.360215    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.360918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.360890    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.362597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.362563    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.386673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.386624    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwq6\" (UniqueName: \"kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6\") pod \"maas-api-revoke-sub-04b06ea6-tmpqz\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.436683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.436659    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:07:58.742883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.742851    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"]
Nov 02 15:07:58.744187 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:07:58.744162    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501afba0_6ba3_4977_bc41_aadc514aa6d6.slice/crio-a695144a3ab43949a7d85e2e14db1df8f36fb0febee208416f278c492ea70d76 WatchSource:0}: Error finding container a695144a3ab43949a7d85e2e14db1df8f36fb0febee208416f278c492ea70d76: Status 404 returned error can't find the container with id a695144a3ab43949a7d85e2e14db1df8f36fb0febee208416f278c492ea70d76
Nov 02 15:07:58.746083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:58.746050    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:07:59.167647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:07:59.167558    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz" event={"ID":"501afba0-6ba3-4977-bc41-aadc514aa6d6","Type":"ContainerStarted","Data":"a695144a3ab43949a7d85e2e14db1df8f36fb0febee208416f278c492ea70d76"}
Nov 02 15:08:01.020886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.020850    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"]
Nov 02 15:08:01.040229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.040200    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.045129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.045072    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"]
Nov 02 15:08:01.188248 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.188217    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pfc\" (UniqueName: \"kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.188383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.188251    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.188454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.188411    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.289781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.289725    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pfc\" (UniqueName: \"kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.289781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.289757    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.289918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.289821    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.290452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.290431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.291966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.291949    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.311906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.311884    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pfc\" (UniqueName: \"kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc\") pod \"maas-api-revoke-sub-649a2b12-zn2xg\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.364021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.364000    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:01.663316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:01.663287    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"]
Nov 02 15:08:01.664461 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:01.664431    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c62b5a7_f8e0_4c03_9970_b1aa07f4d9af.slice/crio-11fa49df4649206815b1bd4c74db595d0877e52b20c04e2b8a6f043f806531f2 WatchSource:0}: Error finding container 11fa49df4649206815b1bd4c74db595d0877e52b20c04e2b8a6f043f806531f2: Status 404 returned error can't find the container with id 11fa49df4649206815b1bd4c74db595d0877e52b20c04e2b8a6f043f806531f2
Nov 02 15:08:02.190787 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:02.190707    5041 generic.go:356] "Generic (PLEG): container finished" podID="501afba0-6ba3-4977-bc41-aadc514aa6d6" containerID="1d2e11e7da35a22523ac0ebc21b8bafa0d253f1c0691fd5cc4b9a958d4cdea7f" exitCode=0
Nov 02 15:08:02.191178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:02.190786    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz" event={"ID":"501afba0-6ba3-4977-bc41-aadc514aa6d6","Type":"ContainerDied","Data":"1d2e11e7da35a22523ac0ebc21b8bafa0d253f1c0691fd5cc4b9a958d4cdea7f"}
Nov 02 15:08:02.193012 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:02.192984    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg" event={"ID":"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af","Type":"ContainerStarted","Data":"2cddd13831f3395c35692a1c6af64ae650bf3ec8812488708cc53678611d5af7"}
Nov 02 15:08:02.193122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:02.193036    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg" event={"ID":"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af","Type":"ContainerStarted","Data":"11fa49df4649206815b1bd4c74db595d0877e52b20c04e2b8a6f043f806531f2"}
Nov 02 15:08:02.270666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:02.270610    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg" podStartSLOduration=2.270595287 podStartE2EDuration="2.270595287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:02.268509473 +0000 UTC m=+734.526578924" watchObservedRunningTime="2026-11-02 15:08:02.270595287 +0000 UTC m=+734.528664737"
Nov 02 15:08:03.200880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.200844    5041 generic.go:356] "Generic (PLEG): container finished" podID="9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" containerID="2cddd13831f3395c35692a1c6af64ae650bf3ec8812488708cc53678611d5af7" exitCode=0
Nov 02 15:08:03.201263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.200931    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg" event={"ID":"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af","Type":"ContainerDied","Data":"2cddd13831f3395c35692a1c6af64ae650bf3ec8812488708cc53678611d5af7"}
Nov 02 15:08:03.355855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.355832    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:08:03.520538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.520461    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca\") pod \"501afba0-6ba3-4977-bc41-aadc514aa6d6\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") "
Nov 02 15:08:03.520538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.520519    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6\" (UniqueName: \"kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6\") pod \"501afba0-6ba3-4977-bc41-aadc514aa6d6\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") "
Nov 02 15:08:03.520538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.520543    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca\") pod \"501afba0-6ba3-4977-bc41-aadc514aa6d6\" (UID: \"501afba0-6ba3-4977-bc41-aadc514aa6d6\") "
Nov 02 15:08:03.520824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.520801    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca" pod "501afba0-6ba3-4977-bc41-aadc514aa6d6" (UID: "501afba0-6ba3-4977-bc41-aadc514aa6d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:03.522685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.522654    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6" pod "501afba0-6ba3-4977-bc41-aadc514aa6d6" (UID: "501afba0-6ba3-4977-bc41-aadc514aa6d6"). InnerVolumeSpecName "kube-api-access-8jwq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:03.522855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.522834    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca" pod "501afba0-6ba3-4977-bc41-aadc514aa6d6" (UID: "501afba0-6ba3-4977-bc41-aadc514aa6d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:03.622261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.622228    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:03.622261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.622255    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwq6\" (UniqueName: \"kubernetes.io/projected/501afba0-6ba3-4977-bc41-aadc514aa6d6-kube-api-access-8jwq6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:03.622261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:03.622266    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501afba0-6ba3-4977-bc41-aadc514aa6d6-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:04.208480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.208455    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"
Nov 02 15:08:04.208870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.208488    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz" event={"ID":"501afba0-6ba3-4977-bc41-aadc514aa6d6","Type":"ContainerDied","Data":"a695144a3ab43949a7d85e2e14db1df8f36fb0febee208416f278c492ea70d76"}
Nov 02 15:08:04.208870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.208540    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a695144a3ab43949a7d85e2e14db1df8f36fb0febee208416f278c492ea70d76"
Nov 02 15:08:04.346092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.346072    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:04.530424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.530314    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca\") pod \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") "
Nov 02 15:08:04.530424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.530395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc\" (UniqueName: \"kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc\") pod \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") "
Nov 02 15:08:04.530649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.530493    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca\") pod \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\" (UID: \"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af\") "
Nov 02 15:08:04.530882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.530855    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca" pod "9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" (UID: "9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:04.532456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.532429    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc" pod "9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" (UID: "9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af"). InnerVolumeSpecName "kube-api-access-m4pfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:04.532456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.532438    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca" pod "9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" (UID: "9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:04.631926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.631889    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:04.631926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.631924    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pfc\" (UniqueName: \"kubernetes.io/projected/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-kube-api-access-m4pfc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:04.632062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:04.631936    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:05.215765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:05.215732    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg" event={"ID":"9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af","Type":"ContainerDied","Data":"11fa49df4649206815b1bd4c74db595d0877e52b20c04e2b8a6f043f806531f2"}
Nov 02 15:08:05.216128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:05.215783    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11fa49df4649206815b1bd4c74db595d0877e52b20c04e2b8a6f043f806531f2"
Nov 02 15:08:05.216128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:05.215750    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"
Nov 02 15:08:06.769218 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:08:06.769168    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:08:07.900285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.900249    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"]
Nov 02 15:08:07.901015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.900996    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" containerName="revoke-keys"
Nov 02 15:08:07.901062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.901018    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" containerName="revoke-keys"
Nov 02 15:08:07.901062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.901054    5041 cpu_manager.go:414] "containerMap: removing container" podUID="501afba0-6ba3-4977-bc41-aadc514aa6d6" containerName="revoke-keys"
Nov 02 15:08:07.901128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.901062    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501afba0-6ba3-4977-bc41-aadc514aa6d6" containerName="revoke-keys"
Nov 02 15:08:07.901229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.901218    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" containerName="revoke-keys"
Nov 02 15:08:07.901264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.901237    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="501afba0-6ba3-4977-bc41-aadc514aa6d6" containerName="revoke-keys"
Nov 02 15:08:07.904332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.904315    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:07.929954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.929918    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:08:07.930089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.929970    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:08:07.934181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:07.934156    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"]
Nov 02 15:08:08.065531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.065503    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.065716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.065556    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.065716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.065574    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtqr\" (UniqueName: \"kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.166692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.166630    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.166692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.166677    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.166855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.166694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtqr\" (UniqueName: \"kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.167369 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.167346    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.170988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.170960    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.195049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.195025    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtqr\" (UniqueName: \"kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr\") pod \"maas-api-revoke-sub-f79317dd-x8cjp\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.218263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.218233    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:08.538016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:08.537987    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"]
Nov 02 15:08:08.539078 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:08.539048    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7965717_df85_4506_a64a_a873f1825972.slice/crio-197d05dd2c3d4fa9e4f1bde4072f9024b4a88d264dca6df88173e3a81be63712 WatchSource:0}: Error finding container 197d05dd2c3d4fa9e4f1bde4072f9024b4a88d264dca6df88173e3a81be63712: Status 404 returned error can't find the container with id 197d05dd2c3d4fa9e4f1bde4072f9024b4a88d264dca6df88173e3a81be63712
Nov 02 15:08:09.251956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:09.251926    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp" event={"ID":"f7965717-df85-4506-a64a-a873f1825972","Type":"ContainerStarted","Data":"7d03f81f9da98a546be10e3d02407528e3c72b71f502a6a9d35d15d110022a59"}
Nov 02 15:08:09.252331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:09.251972    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp" event={"ID":"f7965717-df85-4506-a64a-a873f1825972","Type":"ContainerStarted","Data":"197d05dd2c3d4fa9e4f1bde4072f9024b4a88d264dca6df88173e3a81be63712"}
Nov 02 15:08:09.301049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:09.301008    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp" podStartSLOduration=2.30099478 podStartE2EDuration="2.30099478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:09.299600458 +0000 UTC m=+741.557669931" watchObservedRunningTime="2026-11-02 15:08:09.30099478 +0000 UTC m=+741.559064229"
Nov 02 15:08:10.259809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:10.259772    5041 generic.go:356] "Generic (PLEG): container finished" podID="f7965717-df85-4506-a64a-a873f1825972" containerID="7d03f81f9da98a546be10e3d02407528e3c72b71f502a6a9d35d15d110022a59" exitCode=0
Nov 02 15:08:10.260199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:10.259846    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp" event={"ID":"f7965717-df85-4506-a64a-a873f1825972","Type":"ContainerDied","Data":"7d03f81f9da98a546be10e3d02407528e3c72b71f502a6a9d35d15d110022a59"}
Nov 02 15:08:11.419232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.419204    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:11.498216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.498190    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr\" (UniqueName: \"kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr\") pod \"f7965717-df85-4506-a64a-a873f1825972\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") "
Nov 02 15:08:11.498395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.498239    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca\") pod \"f7965717-df85-4506-a64a-a873f1825972\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") "
Nov 02 15:08:11.498395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.498312    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca\") pod \"f7965717-df85-4506-a64a-a873f1825972\" (UID: \"f7965717-df85-4506-a64a-a873f1825972\") "
Nov 02 15:08:11.498735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.498701    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca" pod "f7965717-df85-4506-a64a-a873f1825972" (UID: "f7965717-df85-4506-a64a-a873f1825972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:11.500379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.500341    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr" pod "f7965717-df85-4506-a64a-a873f1825972" (UID: "f7965717-df85-4506-a64a-a873f1825972"). InnerVolumeSpecName "kube-api-access-2vtqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:11.500475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.500421    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca" pod "f7965717-df85-4506-a64a-a873f1825972" (UID: "f7965717-df85-4506-a64a-a873f1825972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:11.599723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.599670    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtqr\" (UniqueName: \"kubernetes.io/projected/f7965717-df85-4506-a64a-a873f1825972-kube-api-access-2vtqr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:11.599723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.599696    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7965717-df85-4506-a64a-a873f1825972-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:11.599723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:11.599705    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7965717-df85-4506-a64a-a873f1825972-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:12.277952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.277919    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"
Nov 02 15:08:12.277952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.277939    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp" event={"ID":"f7965717-df85-4506-a64a-a873f1825972","Type":"ContainerDied","Data":"197d05dd2c3d4fa9e4f1bde4072f9024b4a88d264dca6df88173e3a81be63712"}
Nov 02 15:08:12.278135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.277989    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="197d05dd2c3d4fa9e4f1bde4072f9024b4a88d264dca6df88173e3a81be63712"
Nov 02 15:08:12.973303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.973249    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"]
Nov 02 15:08:12.973987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.973967    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f7965717-df85-4506-a64a-a873f1825972" containerName="revoke-keys"
Nov 02 15:08:12.974047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.973990    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7965717-df85-4506-a64a-a873f1825972" containerName="revoke-keys"
Nov 02 15:08:12.974137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.974126    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7965717-df85-4506-a64a-a873f1825972" containerName="revoke-keys"
Nov 02 15:08:12.977045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:12.977027    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.002706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.002683    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:08:13.002706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.002692    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:08:13.006229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.006204    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"]
Nov 02 15:08:13.013607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.013582    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jb5\" (UniqueName: \"kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.013727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.013709    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.013864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.013740    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.114423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.114389    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jb5\" (UniqueName: \"kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.114564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.114500    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.114564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.114531    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.115195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.115175    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.116897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.116875    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.137036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.137014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jb5\" (UniqueName: \"kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5\") pod \"maas-api-revoke-sub-f6d970ba-tnbvd\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.289336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.289266    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:13.589934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:13.589898    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"]
Nov 02 15:08:13.590803 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:13.590778    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf760fc72_2e8b_484a_955a_35a029acc55b.slice/crio-5d5072483644514fe5e40232ffd86f72033db6bea101329fa9be0577df902e83 WatchSource:0}: Error finding container 5d5072483644514fe5e40232ffd86f72033db6bea101329fa9be0577df902e83: Status 404 returned error can't find the container with id 5d5072483644514fe5e40232ffd86f72033db6bea101329fa9be0577df902e83
Nov 02 15:08:14.290854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:14.290759    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd" event={"ID":"f760fc72-2e8b-484a-955a-35a029acc55b","Type":"ContainerStarted","Data":"1385d820c5619e5611371475cfb7f6879119c5e5adc04e741a6d3a3dc5d1f85d"}
Nov 02 15:08:14.290854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:14.290822    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd" event={"ID":"f760fc72-2e8b-484a-955a-35a029acc55b","Type":"ContainerStarted","Data":"5d5072483644514fe5e40232ffd86f72033db6bea101329fa9be0577df902e83"}
Nov 02 15:08:14.332634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:14.332587    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd" podStartSLOduration=2.3325735180000002 podStartE2EDuration="2.332573518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:14.328816184 +0000 UTC m=+746.586885634" watchObservedRunningTime="2026-11-02 15:08:14.332573518 +0000 UTC m=+746.590642968"
Nov 02 15:08:15.298802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:15.298762    5041 generic.go:356] "Generic (PLEG): container finished" podID="f760fc72-2e8b-484a-955a-35a029acc55b" containerID="1385d820c5619e5611371475cfb7f6879119c5e5adc04e741a6d3a3dc5d1f85d" exitCode=0
Nov 02 15:08:15.299178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:15.298825    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd" event={"ID":"f760fc72-2e8b-484a-955a-35a029acc55b","Type":"ContainerDied","Data":"1385d820c5619e5611371475cfb7f6879119c5e5adc04e741a6d3a3dc5d1f85d"}
Nov 02 15:08:16.448863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.448840    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:16.547086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.547054    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca\") pod \"f760fc72-2e8b-484a-955a-35a029acc55b\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") "
Nov 02 15:08:16.547260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.547213    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5\" (UniqueName: \"kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5\") pod \"f760fc72-2e8b-484a-955a-35a029acc55b\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") "
Nov 02 15:08:16.547342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.547270    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca\") pod \"f760fc72-2e8b-484a-955a-35a029acc55b\" (UID: \"f760fc72-2e8b-484a-955a-35a029acc55b\") "
Nov 02 15:08:16.547818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.547783    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca" pod "f760fc72-2e8b-484a-955a-35a029acc55b" (UID: "f760fc72-2e8b-484a-955a-35a029acc55b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:16.549464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.549435    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca" pod "f760fc72-2e8b-484a-955a-35a029acc55b" (UID: "f760fc72-2e8b-484a-955a-35a029acc55b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:16.549547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.549441    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5" pod "f760fc72-2e8b-484a-955a-35a029acc55b" (UID: "f760fc72-2e8b-484a-955a-35a029acc55b"). InnerVolumeSpecName "kube-api-access-z8jb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:16.648824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.648792    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:16.648824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.648819    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jb5\" (UniqueName: \"kubernetes.io/projected/f760fc72-2e8b-484a-955a-35a029acc55b-kube-api-access-z8jb5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:16.649020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:16.648842    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f760fc72-2e8b-484a-955a-35a029acc55b-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:17.312951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:17.312919    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"
Nov 02 15:08:17.313166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:17.312945    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd" event={"ID":"f760fc72-2e8b-484a-955a-35a029acc55b","Type":"ContainerDied","Data":"5d5072483644514fe5e40232ffd86f72033db6bea101329fa9be0577df902e83"}
Nov 02 15:08:17.313166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:17.313025    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5072483644514fe5e40232ffd86f72033db6bea101329fa9be0577df902e83"
Nov 02 15:08:22.379523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.379494    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"]
Nov 02 15:08:22.380032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.380017    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f760fc72-2e8b-484a-955a-35a029acc55b" containerName="revoke-keys"
Nov 02 15:08:22.380169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.380033    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f760fc72-2e8b-484a-955a-35a029acc55b" containerName="revoke-keys"
Nov 02 15:08:22.380169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.380138    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f760fc72-2e8b-484a-955a-35a029acc55b" containerName="revoke-keys"
Nov 02 15:08:22.383076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.383055    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.397093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.397073    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:08:22.408427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.408393    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:08:22.413577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.413555    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"]
Nov 02 15:08:22.508880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.508849    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.508994 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.508897    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.509042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.508996    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzssv\" (UniqueName: \"kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.610291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.610265    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzssv\" (UniqueName: \"kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.610430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.610348    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.610430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.610385    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.611023 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.611004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.612695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.612675    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.636688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.636634    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzssv\" (UniqueName: \"kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv\") pod \"maas-api-revoke-sub-a3140768-nd8dm\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.695168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.695147    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:22.992504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:22.992476    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"]
Nov 02 15:08:22.994588 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:22.994561    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccff4436_d16c_4f4a_ab8c_b7f473c1a5b8.slice/crio-79b1c2a0919b8178c26ac99e8e2227862a60184572ee94483524e336ae586be4 WatchSource:0}: Error finding container 79b1c2a0919b8178c26ac99e8e2227862a60184572ee94483524e336ae586be4: Status 404 returned error can't find the container with id 79b1c2a0919b8178c26ac99e8e2227862a60184572ee94483524e336ae586be4
Nov 02 15:08:23.339824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.339677    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"]
Nov 02 15:08:23.346624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.345195    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.356770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.356721    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm" event={"ID":"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8","Type":"ContainerStarted","Data":"79b1c2a0919b8178c26ac99e8e2227862a60184572ee94483524e336ae586be4"}
Nov 02 15:08:23.365664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.365642    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ecfe1-openshift-default-dockercfg-gg2h7\""
Nov 02 15:08:23.379213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.379189    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"]
Nov 02 15:08:23.418928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.418905    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.418944    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419068    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419173    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419203    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6ps\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419450    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419499    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.419566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.419526    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520234    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6ps\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520318    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520340    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520356    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520444    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.520757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520604    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.521098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.520961    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.521098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.521000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.521098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.521037    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.521251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.521231    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.521353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.521336    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.522564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.522544    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.522798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.522782    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.542518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.542496    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6ps\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.542518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.542512    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.661108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.661074    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:23.985752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:23.985726    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"]
Nov 02 15:08:24.009186 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:24.009151    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d386b5_12f7_4590_8270_51204c1da50e.slice/crio-5c87c9d7f0206aa97aa93d8f4c11addb35d9de12151c2f4a7d88af03fa7171ab WatchSource:0}: Error finding container 5c87c9d7f0206aa97aa93d8f4c11addb35d9de12151c2f4a7d88af03fa7171ab: Status 404 returned error can't find the container with id 5c87c9d7f0206aa97aa93d8f4c11addb35d9de12151c2f4a7d88af03fa7171ab
Nov 02 15:08:24.011562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.011530    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:24.011637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.011599    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:24.011685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.011640    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:24.365244 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.365166    5041 generic.go:356] "Generic (PLEG): container finished" podID="ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" containerID="57caa320cff8c80596a56dc2b24c9581b4c2d42057f8178843521cf089ab5fe9" exitCode=0
Nov 02 15:08:24.365408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.365247    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm" event={"ID":"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8","Type":"ContainerDied","Data":"57caa320cff8c80596a56dc2b24c9581b4c2d42057f8178843521cf089ab5fe9"}
Nov 02 15:08:24.367021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.366986    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" event={"ID":"91d386b5-12f7-4590-8270-51204c1da50e","Type":"ContainerStarted","Data":"cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2"}
Nov 02 15:08:24.367143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.367033    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" event={"ID":"91d386b5-12f7-4590-8270-51204c1da50e","Type":"ContainerStarted","Data":"5c87c9d7f0206aa97aa93d8f4c11addb35d9de12151c2f4a7d88af03fa7171ab"}
Nov 02 15:08:24.461908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.461864    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podStartSLOduration=1.4618505 podStartE2EDuration="1.4618505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:24.461036083 +0000 UTC m=+756.719105534" watchObservedRunningTime="2026-11-02 15:08:24.4618505 +0000 UTC m=+756.719919950"
Nov 02 15:08:24.661483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:24.661413    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:25.606943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.606914    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:25.662902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.662825    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:08:25.662902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.662881    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 15:08:25.683033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.682997    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca\") pod \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") "
Nov 02 15:08:25.683183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.683088    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv\" (UniqueName: \"kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv\") pod \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") "
Nov 02 15:08:25.683183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.683160    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca\") pod \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\" (UID: \"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8\") "
Nov 02 15:08:25.683453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.683427    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca" pod "ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" (UID: "ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:25.683653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.683625    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:25.685253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.685224    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca" pod "ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" (UID: "ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:25.685253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.685233    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv" pod "ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" (UID: "ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8"). InnerVolumeSpecName "kube-api-access-fzssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:25.785230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.785200    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzssv\" (UniqueName: \"kubernetes.io/projected/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-kube-api-access-fzssv\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:25.785230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:25.785231    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:26.194736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.194704    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"]
Nov 02 15:08:26.195296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.195277    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" containerName="revoke-keys"
Nov 02 15:08:26.195296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.195294    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" containerName="revoke-keys"
Nov 02 15:08:26.195513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.195407    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" containerName="revoke-keys"
Nov 02 15:08:26.201338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.198519    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.257393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.255132    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"]
Nov 02 15:08:26.290480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.290450    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbpv\" (UniqueName: \"kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.290612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.290501    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.290654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.290635    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.382018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.381985    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"
Nov 02 15:08:26.382208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.382017    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm" event={"ID":"ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8","Type":"ContainerDied","Data":"79b1c2a0919b8178c26ac99e8e2227862a60184572ee94483524e336ae586be4"}
Nov 02 15:08:26.382208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.382071    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b1c2a0919b8178c26ac99e8e2227862a60184572ee94483524e336ae586be4"
Nov 02 15:08:26.391869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.391841    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.392014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.392001    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.392127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.392111    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbpv\" (UniqueName: \"kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.392581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.392558    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.394602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.394578    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.441333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.441300    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbpv\" (UniqueName: \"kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv\") pod \"maas-api-revoke-sub-b6658ba6-sq77h\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.548221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.548146    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:26.661558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.661521    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:08:26.662024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.661580    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 15:08:26.858784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:26.858751    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"]
Nov 02 15:08:26.859478 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:26.859434    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063e963a_d3a8_438d_91a1_8dd7dd9892f1.slice/crio-ccadd1de4fbe63f510f0276ae342c95a1832e96d4c7d32a94fc4c14fca66c886 WatchSource:0}: Error finding container ccadd1de4fbe63f510f0276ae342c95a1832e96d4c7d32a94fc4c14fca66c886: Status 404 returned error can't find the container with id ccadd1de4fbe63f510f0276ae342c95a1832e96d4c7d32a94fc4c14fca66c886
Nov 02 15:08:27.390988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.390949    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h" event={"ID":"063e963a-d3a8-438d-91a1-8dd7dd9892f1","Type":"ContainerStarted","Data":"e19f2bc5dfc26b62ef2a38a25b7c6178917cdbbc62fd9cd68ac69e986670c792"}
Nov 02 15:08:27.391160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.391028    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h" event={"ID":"063e963a-d3a8-438d-91a1-8dd7dd9892f1","Type":"ContainerStarted","Data":"ccadd1de4fbe63f510f0276ae342c95a1832e96d4c7d32a94fc4c14fca66c886"}
Nov 02 15:08:27.440528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.440465    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h" podStartSLOduration=1.440444744 podStartE2EDuration="1.440444744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:27.430978656 +0000 UTC m=+759.689048106" watchObservedRunningTime="2026-11-02 15:08:27.440444744 +0000 UTC m=+759.698514195"
Nov 02 15:08:27.541481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.541390    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"]
Nov 02 15:08:27.545867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.545846    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.549390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.549348    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"]
Nov 02 15:08:27.606799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.606773    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.606922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.606844    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.606922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.606916    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwjf\" (UniqueName: \"kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.661859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.661829    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:08:27.662225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.661881    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 15:08:27.708071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.708042    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwjf\" (UniqueName: \"kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.708197 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.708140    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.708234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.708203    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.708913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.708894    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.710451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.710432    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.744552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.744525    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwjf\" (UniqueName: \"kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf\") pod \"maas-api-revoke-sub-6078c8d7-nn7lr\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:27.861756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:27.861728    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:28.188336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:28.188310    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"]
Nov 02 15:08:28.189317 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:28.189293    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebacc0eb_768f_426a_8c2a_fc6d12fa44ca.slice/crio-f2acc91f367d452f000710c8505c033b30d6e51bed8eb51de90a05f971d28a29 WatchSource:0}: Error finding container f2acc91f367d452f000710c8505c033b30d6e51bed8eb51de90a05f971d28a29: Status 404 returned error can't find the container with id f2acc91f367d452f000710c8505c033b30d6e51bed8eb51de90a05f971d28a29
Nov 02 15:08:28.398567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:28.398531    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr" event={"ID":"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca","Type":"ContainerStarted","Data":"f2acc91f367d452f000710c8505c033b30d6e51bed8eb51de90a05f971d28a29"}
Nov 02 15:08:28.400245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:28.400214    5041 generic.go:356] "Generic (PLEG): container finished" podID="063e963a-d3a8-438d-91a1-8dd7dd9892f1" containerID="e19f2bc5dfc26b62ef2a38a25b7c6178917cdbbc62fd9cd68ac69e986670c792" exitCode=0
Nov 02 15:08:28.400400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:28.400300    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h" event={"ID":"063e963a-d3a8-438d-91a1-8dd7dd9892f1","Type":"ContainerDied","Data":"e19f2bc5dfc26b62ef2a38a25b7c6178917cdbbc62fd9cd68ac69e986670c792"}
Nov 02 15:08:28.661964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:28.661938    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:08:28.662259 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:28.661978    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 15:08:29.340607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.340573    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"]
Nov 02 15:08:29.344113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.344093    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.366781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.366754    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"]
Nov 02 15:08:29.410540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.410513    5041 generic.go:356] "Generic (PLEG): container finished" podID="ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" containerID="0788a001b50585f91bd8e84401979ee1fe0d724adea3cde4abca9fb3032deaeb" exitCode=0
Nov 02 15:08:29.410689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.410605    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr" event={"ID":"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca","Type":"ContainerDied","Data":"0788a001b50585f91bd8e84401979ee1fe0d724adea3cde4abca9fb3032deaeb"}
Nov 02 15:08:29.468904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.468874    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4t2g\" (UniqueName: \"kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.469088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.468990    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.469088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.469034    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.570046 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.570012    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.570237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.570061    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.570237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.570217    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4t2g\" (UniqueName: \"kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.570904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.570874    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.572590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.572562    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.592988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.592969    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4t2g\" (UniqueName: \"kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g\") pod \"maas-api-revoke-sub-bcc270cc-r4dj5\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.636386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.632523    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:29.656150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.656125    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:29.662289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.662260    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:08:29.662660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.662297    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 15:08:29.772723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.772697    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca\") pod \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") "
Nov 02 15:08:29.772859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.772773    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv\" (UniqueName: \"kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv\") pod \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") "
Nov 02 15:08:29.772859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.772828    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca\") pod \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\" (UID: \"063e963a-d3a8-438d-91a1-8dd7dd9892f1\") "
Nov 02 15:08:29.773550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.773518    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca" pod "063e963a-d3a8-438d-91a1-8dd7dd9892f1" (UID: "063e963a-d3a8-438d-91a1-8dd7dd9892f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:29.777400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.776148    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv" pod "063e963a-d3a8-438d-91a1-8dd7dd9892f1" (UID: "063e963a-d3a8-438d-91a1-8dd7dd9892f1"). InnerVolumeSpecName "kube-api-access-hnbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:29.777400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.776433    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca" pod "063e963a-d3a8-438d-91a1-8dd7dd9892f1" (UID: "063e963a-d3a8-438d-91a1-8dd7dd9892f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:29.874184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.874108    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:29.874184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.874146    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbpv\" (UniqueName: \"kubernetes.io/projected/063e963a-d3a8-438d-91a1-8dd7dd9892f1-kube-api-access-hnbpv\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:29.874184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.874156    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e963a-d3a8-438d-91a1-8dd7dd9892f1-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:29.990638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:29.990602    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"]
Nov 02 15:08:29.992019 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:29.991989    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cb1c5b_d0e6_45b0_a5a1_47ff4d397d8e.slice/crio-b90b5909f86f47d9fa680af031b86a33fcf224a72ab53c9b6899c22e0fb1043d WatchSource:0}: Error finding container b90b5909f86f47d9fa680af031b86a33fcf224a72ab53c9b6899c22e0fb1043d: Status 404 returned error can't find the container with id b90b5909f86f47d9fa680af031b86a33fcf224a72ab53c9b6899c22e0fb1043d
Nov 02 15:08:30.420054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.420020    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5" event={"ID":"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e","Type":"ContainerStarted","Data":"b90b5909f86f47d9fa680af031b86a33fcf224a72ab53c9b6899c22e0fb1043d"}
Nov 02 15:08:30.421482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.421457    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"
Nov 02 15:08:30.421621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.421489    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h" event={"ID":"063e963a-d3a8-438d-91a1-8dd7dd9892f1","Type":"ContainerDied","Data":"ccadd1de4fbe63f510f0276ae342c95a1832e96d4c7d32a94fc4c14fca66c886"}
Nov 02 15:08:30.421621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.421535    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccadd1de4fbe63f510f0276ae342c95a1832e96d4c7d32a94fc4c14fca66c886"
Nov 02 15:08:30.566036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.566011    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:30.582668 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.582617    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca\") pod \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") "
Nov 02 15:08:30.583030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.583005    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca" pod "ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" (UID: "ebacc0eb-768f-426a-8c2a-fc6d12fa44ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:30.683752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.683718    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca\") pod \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") "
Nov 02 15:08:30.683752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.683757    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf\" (UniqueName: \"kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf\") pod \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\" (UID: \"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca\") "
Nov 02 15:08:30.684237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.684150    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:30.685989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.685964    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf" pod "ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" (UID: "ebacc0eb-768f-426a-8c2a-fc6d12fa44ca"). InnerVolumeSpecName "kube-api-access-zvwjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:30.686121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.686023    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca" pod "ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" (UID: "ebacc0eb-768f-426a-8c2a-fc6d12fa44ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:30.785021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.784988    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:30.785179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.785026    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwjf\" (UniqueName: \"kubernetes.io/projected/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca-kube-api-access-zvwjf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:30.931990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.931961    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:30.932827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.932772    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:30.933600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.933573    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:08:30.968960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.968935    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"]
Nov 02 15:08:30.969513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.969496    5041 cpu_manager.go:414] "containerMap: removing container" podUID="063e963a-d3a8-438d-91a1-8dd7dd9892f1" containerName="revoke-keys"
Nov 02 15:08:30.969588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.969513    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063e963a-d3a8-438d-91a1-8dd7dd9892f1" containerName="revoke-keys"
Nov 02 15:08:30.969588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.969547    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" containerName="revoke-keys"
Nov 02 15:08:30.969588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.969555    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" containerName="revoke-keys"
Nov 02 15:08:30.969703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.969647    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" containerName="revoke-keys"
Nov 02 15:08:30.969703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.969658    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="063e963a-d3a8-438d-91a1-8dd7dd9892f1" containerName="revoke-keys"
Nov 02 15:08:30.973156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.973139    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:30.986965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.986941    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:30.987059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.987000    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:30.987324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.987302    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnqwl\" (UniqueName: \"kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:30.994520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:30.994496    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"]
Nov 02 15:08:31.088470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.088434    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.088659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.088639    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnqwl\" (UniqueName: \"kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.088717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.088702    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.089523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.089498    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.090712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.090687    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.111723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.111700    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnqwl\" (UniqueName: \"kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl\") pod \"maas-api-revoke-sub-245255d5-hccgd\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.288861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.288830    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:31.429741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.429711    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"
Nov 02 15:08:31.429910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.429711    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr" event={"ID":"ebacc0eb-768f-426a-8c2a-fc6d12fa44ca","Type":"ContainerDied","Data":"f2acc91f367d452f000710c8505c033b30d6e51bed8eb51de90a05f971d28a29"}
Nov 02 15:08:31.429910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.429831    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2acc91f367d452f000710c8505c033b30d6e51bed8eb51de90a05f971d28a29"
Nov 02 15:08:31.431345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.431308    5041 generic.go:356] "Generic (PLEG): container finished" podID="91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" containerID="8a4fe7f09424dc5e03927f38e5678c9bdf57fb2352299c9869152813ff642d87" exitCode=0
Nov 02 15:08:31.431480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.431394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5" event={"ID":"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e","Type":"ContainerDied","Data":"8a4fe7f09424dc5e03927f38e5678c9bdf57fb2352299c9869152813ff642d87"}
Nov 02 15:08:31.588612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:31.588586    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"]
Nov 02 15:08:31.589735 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:31.589707    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcba7116_075b_4789_aecf_774639e6757c.slice/crio-8d1509ae1083e0d5949d33dc656a207c9080ce2e9e772ec7589af4215f490027 WatchSource:0}: Error finding container 8d1509ae1083e0d5949d33dc656a207c9080ce2e9e772ec7589af4215f490027: Status 404 returned error can't find the container with id 8d1509ae1083e0d5949d33dc656a207c9080ce2e9e772ec7589af4215f490027
Nov 02 15:08:32.440411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.440374    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd" event={"ID":"fcba7116-075b-4789-aecf-774639e6757c","Type":"ContainerStarted","Data":"5101c868f670a042a09e4dbe581ada8861ea1b930ad7dd91dac94a1c4ab647f5"}
Nov 02 15:08:32.440816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.440431    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd" event={"ID":"fcba7116-075b-4789-aecf-774639e6757c","Type":"ContainerStarted","Data":"8d1509ae1083e0d5949d33dc656a207c9080ce2e9e772ec7589af4215f490027"}
Nov 02 15:08:32.483705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.483638    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd" podStartSLOduration=2.48361335 podStartE2EDuration="2.48361335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:32.479706895 +0000 UTC m=+764.737776346" watchObservedRunningTime="2026-11-02 15:08:32.48361335 +0000 UTC m=+764.741682802"
Nov 02 15:08:32.610844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.610819    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:32.709609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.709541    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g\" (UniqueName: \"kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g\") pod \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") "
Nov 02 15:08:32.709609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.709576    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca\") pod \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") "
Nov 02 15:08:32.709609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.709605    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca\") pod \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\" (UID: \"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e\") "
Nov 02 15:08:32.709979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.709955    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca" pod "91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" (UID: "91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:32.710066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.710053    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:32.711712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.711685    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g" pod "91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" (UID: "91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e"). InnerVolumeSpecName "kube-api-access-l4t2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:32.711818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.711717    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca" pod "91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" (UID: "91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:32.810800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.810770    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4t2g\" (UniqueName: \"kubernetes.io/projected/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-kube-api-access-l4t2g\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:32.810800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:32.810800    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:33.450513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:33.450479    5041 generic.go:356] "Generic (PLEG): container finished" podID="fcba7116-075b-4789-aecf-774639e6757c" containerID="5101c868f670a042a09e4dbe581ada8861ea1b930ad7dd91dac94a1c4ab647f5" exitCode=0
Nov 02 15:08:33.450944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:33.450567    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd" event={"ID":"fcba7116-075b-4789-aecf-774639e6757c","Type":"ContainerDied","Data":"5101c868f670a042a09e4dbe581ada8861ea1b930ad7dd91dac94a1c4ab647f5"}
Nov 02 15:08:33.451941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:33.451913    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"
Nov 02 15:08:33.451941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:33.451931    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5" event={"ID":"91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e","Type":"ContainerDied","Data":"b90b5909f86f47d9fa680af031b86a33fcf224a72ab53c9b6899c22e0fb1043d"}
Nov 02 15:08:33.452083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:33.451969    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90b5909f86f47d9fa680af031b86a33fcf224a72ab53c9b6899c22e0fb1043d"
Nov 02 15:08:34.626040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.626017    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:34.731893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.731826    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca\") pod \"fcba7116-075b-4789-aecf-774639e6757c\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") "
Nov 02 15:08:34.732027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.731919    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl\" (UniqueName: \"kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl\") pod \"fcba7116-075b-4789-aecf-774639e6757c\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") "
Nov 02 15:08:34.732027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.731961    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca\") pod \"fcba7116-075b-4789-aecf-774639e6757c\" (UID: \"fcba7116-075b-4789-aecf-774639e6757c\") "
Nov 02 15:08:34.732241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.732218    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca" pod "fcba7116-075b-4789-aecf-774639e6757c" (UID: "fcba7116-075b-4789-aecf-774639e6757c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:34.732442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.732423    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcba7116-075b-4789-aecf-774639e6757c-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:34.734138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.734116    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl" pod "fcba7116-075b-4789-aecf-774639e6757c" (UID: "fcba7116-075b-4789-aecf-774639e6757c"). InnerVolumeSpecName "kube-api-access-qnqwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:34.734207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.734160    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca" pod "fcba7116-075b-4789-aecf-774639e6757c" (UID: "fcba7116-075b-4789-aecf-774639e6757c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:34.833394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.833349    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnqwl\" (UniqueName: \"kubernetes.io/projected/fcba7116-075b-4789-aecf-774639e6757c-kube-api-access-qnqwl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:34.833394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:34.833391    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcba7116-075b-4789-aecf-774639e6757c-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:35.467306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:35.467272    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd" event={"ID":"fcba7116-075b-4789-aecf-774639e6757c","Type":"ContainerDied","Data":"8d1509ae1083e0d5949d33dc656a207c9080ce2e9e772ec7589af4215f490027"}
Nov 02 15:08:35.467505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:35.467326    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d1509ae1083e0d5949d33dc656a207c9080ce2e9e772ec7589af4215f490027"
Nov 02 15:08:35.467505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:35.467283    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"
Nov 02 15:08:36.019268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.019233    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"]
Nov 02 15:08:36.019871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.019855    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fcba7116-075b-4789-aecf-774639e6757c" containerName="revoke-keys"
Nov 02 15:08:36.019913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.019873    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcba7116-075b-4789-aecf-774639e6757c" containerName="revoke-keys"
Nov 02 15:08:36.019913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.019906    5041 cpu_manager.go:414] "containerMap: removing container" podUID="91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" containerName="revoke-keys"
Nov 02 15:08:36.019913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.019912    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" containerName="revoke-keys"
Nov 02 15:08:36.020013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.019993    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" containerName="revoke-keys"
Nov 02 15:08:36.020013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.020007    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcba7116-075b-4789-aecf-774639e6757c" containerName="revoke-keys"
Nov 02 15:08:36.024052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.024033    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.041073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.041053    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:08:36.049264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.049232    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.049546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.049519    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"]
Nov 02 15:08:36.049651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.049523    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpn9b\" (UniqueName: \"kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.049863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.049844    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.059403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.052356    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:08:36.150944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.150911    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.151101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.150966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.151101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.151001    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpn9b\" (UniqueName: \"kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.151609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.151581    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.153216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.153193    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.173980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.173948    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpn9b\" (UniqueName: \"kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b\") pod \"maas-api-revoke-sub-effba133-6hhj8\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.340155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.340084    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:36.641211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:36.641178    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"]
Nov 02 15:08:36.642558 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:36.642521    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf53e0e0_d860_4127_81ef_45a536f76ce3.slice/crio-b6d33125336422a6a879f6ec9a9c5141433a6693297f2a381fcba66f12893991 WatchSource:0}: Error finding container b6d33125336422a6a879f6ec9a9c5141433a6693297f2a381fcba66f12893991: Status 404 returned error can't find the container with id b6d33125336422a6a879f6ec9a9c5141433a6693297f2a381fcba66f12893991
Nov 02 15:08:36.769990 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:08:36.769947    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:08:37.481321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.481287    5041 generic.go:356] "Generic (PLEG): container finished" podID="df53e0e0-d860-4127-81ef-45a536f76ce3" containerID="92774dce9bf0c807faa14e9f579eba3e8e9ef489587ad8d182d6b4d1c1147ceb" exitCode=0
Nov 02 15:08:37.481651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.481346    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8" event={"ID":"df53e0e0-d860-4127-81ef-45a536f76ce3","Type":"ContainerDied","Data":"92774dce9bf0c807faa14e9f579eba3e8e9ef489587ad8d182d6b4d1c1147ceb"}
Nov 02 15:08:37.481651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.481423    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8" event={"ID":"df53e0e0-d860-4127-81ef-45a536f76ce3","Type":"ContainerStarted","Data":"b6d33125336422a6a879f6ec9a9c5141433a6693297f2a381fcba66f12893991"}
Nov 02 15:08:37.658617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.658586    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"]
Nov 02 15:08:37.661902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.661881    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.685297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.685272    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"]
Nov 02 15:08:37.769793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.769761    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.769943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.769908    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx7w\" (UniqueName: \"kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.769988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.769957    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.870844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.870815    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx7w\" (UniqueName: \"kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.870988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.870862    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.870988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.870921    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.871546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.871522    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.873243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.873219    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.895188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.895162    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx7w\" (UniqueName: \"kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w\") pod \"maas-api-revoke-sub-302bb933-gfxrh\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:37.974227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:37.974179    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:38.280765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.280735    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"]
Nov 02 15:08:38.282935 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:38.282902    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf40389_00ec_4183_a747_b4e6b0426b59.slice/crio-7d9678047610fbac5c8eb786f704135cf764da601ac044c78be9a983bf6e14c0 WatchSource:0}: Error finding container 7d9678047610fbac5c8eb786f704135cf764da601ac044c78be9a983bf6e14c0: Status 404 returned error can't find the container with id 7d9678047610fbac5c8eb786f704135cf764da601ac044c78be9a983bf6e14c0
Nov 02 15:08:38.491389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.491322    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh" event={"ID":"1cf40389-00ec-4183-a747-b4e6b0426b59","Type":"ContainerStarted","Data":"7d9678047610fbac5c8eb786f704135cf764da601ac044c78be9a983bf6e14c0"}
Nov 02 15:08:38.495801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.495772    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"]
Nov 02 15:08:38.500093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.500071    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.525834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.525813    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-624254-openshift-default-dockercfg-c82tf\""
Nov 02 15:08:38.530498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.530436    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"]
Nov 02 15:08:38.580116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580089    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580134    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580224    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580285    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580383    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqdd\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580490    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580522    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580601    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.580701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.580644    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681278    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681330    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681377    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681412    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681522    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681556    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681590    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681650    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqdd\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.681814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.681751    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.682298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.682273    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.682478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.682455    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.682652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.682629    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.682795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.682779    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.684204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.684180    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:38.684292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.684274    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.684511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.684493    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.706941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.706915    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.707075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.707056    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqdd\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.782974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.782887    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca\") pod \"df53e0e0-d860-4127-81ef-45a536f76ce3\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") "
Nov 02 15:08:38.783138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.783074    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b\" (UniqueName: \"kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b\") pod \"df53e0e0-d860-4127-81ef-45a536f76ce3\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") "
Nov 02 15:08:38.783202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.783172    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca\") pod \"df53e0e0-d860-4127-81ef-45a536f76ce3\" (UID: \"df53e0e0-d860-4127-81ef-45a536f76ce3\") "
Nov 02 15:08:38.783805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.783775    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca" pod "df53e0e0-d860-4127-81ef-45a536f76ce3" (UID: "df53e0e0-d860-4127-81ef-45a536f76ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:38.786087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.786055    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca" pod "df53e0e0-d860-4127-81ef-45a536f76ce3" (UID: "df53e0e0-d860-4127-81ef-45a536f76ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:38.786187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.786084    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b" pod "df53e0e0-d860-4127-81ef-45a536f76ce3" (UID: "df53e0e0-d860-4127-81ef-45a536f76ce3"). InnerVolumeSpecName "kube-api-access-vpn9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:38.816916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.816885    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:38.884889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.884857    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:38.884983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.884894    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df53e0e0-d860-4127-81ef-45a536f76ce3-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:38.884983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:38.884910    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpn9b\" (UniqueName: \"kubernetes.io/projected/df53e0e0-d860-4127-81ef-45a536f76ce3-kube-api-access-vpn9b\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:39.137413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.137205    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"]
Nov 02 15:08:39.149053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.148998    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:39.149154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.149098    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:39.149154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.149141    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:39.511448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.510179    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" event={"ID":"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd","Type":"ContainerStarted","Data":"657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73"}
Nov 02 15:08:39.511448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.510253    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" event={"ID":"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd","Type":"ContainerStarted","Data":"165c304c57fbb91ac68b23e575833dd5cdb19e161460ba20202540f5fed3f278"}
Nov 02 15:08:39.513666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.513383    5041 generic.go:356] "Generic (PLEG): container finished" podID="1cf40389-00ec-4183-a747-b4e6b0426b59" containerID="98d3ebfe088af0d1e0e1cb9a907cb4568b127ee07aeadd9ef6483703b72bf50b" exitCode=0
Nov 02 15:08:39.513666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.513516    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh" event={"ID":"1cf40389-00ec-4183-a747-b4e6b0426b59","Type":"ContainerDied","Data":"98d3ebfe088af0d1e0e1cb9a907cb4568b127ee07aeadd9ef6483703b72bf50b"}
Nov 02 15:08:39.531395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.522096    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8" event={"ID":"df53e0e0-d860-4127-81ef-45a536f76ce3","Type":"ContainerDied","Data":"b6d33125336422a6a879f6ec9a9c5141433a6693297f2a381fcba66f12893991"}
Nov 02 15:08:39.531395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.522142    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6d33125336422a6a879f6ec9a9c5141433a6693297f2a381fcba66f12893991"
Nov 02 15:08:39.531395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.522202    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"
Nov 02 15:08:39.604848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.604253    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podStartSLOduration=1.604233535 podStartE2EDuration="1.604233535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:39.588814492 +0000 UTC m=+771.846883966" watchObservedRunningTime="2026-11-02 15:08:39.604233535 +0000 UTC m=+771.862302986"
Nov 02 15:08:39.821387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:39.820703    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:40.939411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:40.910918    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:08:40.939411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:40.910977    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:08:40.939411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:40.927751    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"]
Nov 02 15:08:40.939411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:40.928646    5041 cpu_manager.go:414] "containerMap: removing container" podUID="df53e0e0-d860-4127-81ef-45a536f76ce3" containerName="revoke-keys"
Nov 02 15:08:40.939411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:40.928667    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df53e0e0-d860-4127-81ef-45a536f76ce3" containerName="revoke-keys"
Nov 02 15:08:40.939411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:40.928852    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="df53e0e0-d860-4127-81ef-45a536f76ce3" containerName="revoke-keys"
Nov 02 15:08:41.319807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.309885    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"]
Nov 02 15:08:41.319807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.310012    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.386385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.385990    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.386385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.386092    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.386385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.386241    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9rd\" (UniqueName: \"kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.492634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.492468    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9rd\" (UniqueName: \"kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.528057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.496442    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.528057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.526738    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.528057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.527917    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.528057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.513501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.535552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.535522    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.604298    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"]
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.605221    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1cf40389-00ec-4183-a747-b4e6b0426b59" containerName="revoke-keys"
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.605245    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf40389-00ec-4183-a747-b4e6b0426b59" containerName="revoke-keys"
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.623511    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf40389-00ec-4183-a747-b4e6b0426b59" containerName="revoke-keys"
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.628708    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w\" (UniqueName: \"kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w\") pod \"1cf40389-00ec-4183-a747-b4e6b0426b59\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") "
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.628796    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\") pod \"1cf40389-00ec-4183-a747-b4e6b0426b59\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") "
Nov 02 15:08:41.645389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.628977    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca\") pod \"1cf40389-00ec-4183-a747-b4e6b0426b59\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") "
Nov 02 15:08:41.661936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.661884    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca" pod "1cf40389-00ec-4183-a747-b4e6b0426b59" (UID: "1cf40389-00ec-4183-a747-b4e6b0426b59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:41.671998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.671969    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"]
Nov 02 15:08:41.681733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.681681    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh" event={"ID":"1cf40389-00ec-4183-a747-b4e6b0426b59","Type":"ContainerDied","Data":"7d9678047610fbac5c8eb786f704135cf764da601ac044c78be9a983bf6e14c0"}
Nov 02 15:08:41.681932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.681914    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d9678047610fbac5c8eb786f704135cf764da601ac044c78be9a983bf6e14c0"
Nov 02 15:08:41.682072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.672270    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.682258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.680486    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"
Nov 02 15:08:41.728293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.728261    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w" pod "1cf40389-00ec-4183-a747-b4e6b0426b59" (UID: "1cf40389-00ec-4183-a747-b4e6b0426b59"). InnerVolumeSpecName "kube-api-access-jhx7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.769494    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca" pod "1cf40389-00ec-4183-a747-b4e6b0426b59" (UID: "1cf40389-00ec-4183-a747-b4e6b0426b59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.780541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9rd\" (UniqueName: \"kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd\") pod \"maas-api-revoke-sub-6ed65845-449b2\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.783417    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\") pod \"1cf40389-00ec-4183-a747-b4e6b0426b59\" (UID: \"1cf40389-00ec-4183-a747-b4e6b0426b59\") "
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:41.783999    5041 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1cf40389-00ec-4183-a747-b4e6b0426b59/volumes/kubernetes.io~secret/maas-api-xks-ca
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.784017    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca" pod "1cf40389-00ec-4183-a747-b4e6b0426b59" (UID: "1cf40389-00ec-4183-a747-b4e6b0426b59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.784313    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.784479    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.784639    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glxjw\" (UniqueName: \"kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.785386    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhx7w\" (UniqueName: \"kubernetes.io/projected/1cf40389-00ec-4183-a747-b4e6b0426b59-kube-api-access-jhx7w\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.785537    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:41.830446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.785558    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf40389-00ec-4183-a747-b4e6b0426b59-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:41.871884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.841094    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:41.895386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.890129    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.895386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.890207    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glxjw\" (UniqueName: \"kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.895386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.890408    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.937394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.936488    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.953386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.950490    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:08:41.953386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.950554    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:08:41.957812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.957756    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:41.997387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:41.976074    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glxjw\" (UniqueName: \"kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw\") pod \"maas-api-revoke-sub-57202685-4xxkb\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:42.137613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:42.132465    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:08:42.137613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:42.136990    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:42.680401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:42.680359    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"]
Nov 02 15:08:42.681763 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:42.681726    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bff10ba_80e4_4be1_873a_53759ead5447.slice/crio-39eab579df11c7c5293c1411f4080a36ddffd4370746bb25f7e34bca63b9b0a8 WatchSource:0}: Error finding container 39eab579df11c7c5293c1411f4080a36ddffd4370746bb25f7e34bca63b9b0a8: Status 404 returned error can't find the container with id 39eab579df11c7c5293c1411f4080a36ddffd4370746bb25f7e34bca63b9b0a8
Nov 02 15:08:42.751796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:42.751764    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"]
Nov 02 15:08:42.753794 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:42.753771    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e4c139_3746_4611_a926_e1ec04ac43a4.slice/crio-533f67f5a126abc24eadd91c7a84f0105165e8ad511dc169d5d2068ff07528b3 WatchSource:0}: Error finding container 533f67f5a126abc24eadd91c7a84f0105165e8ad511dc169d5d2068ff07528b3: Status 404 returned error can't find the container with id 533f67f5a126abc24eadd91c7a84f0105165e8ad511dc169d5d2068ff07528b3
Nov 02 15:08:42.834390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:42.830652    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:08:42.834390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:42.830720    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:08:43.574910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.574872    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"]
Nov 02 15:08:43.578079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.578057    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.601639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.601614    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"]
Nov 02 15:08:43.692447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.692403    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb" event={"ID":"55e4c139-3746-4611-a926-e1ec04ac43a4","Type":"ContainerStarted","Data":"c6958f09e551b3edd1995fec0c81aa14b18098af75deb455da5f6fbe55e4f7f6"}
Nov 02 15:08:43.692591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.692470    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb" event={"ID":"55e4c139-3746-4611-a926-e1ec04ac43a4","Type":"ContainerStarted","Data":"533f67f5a126abc24eadd91c7a84f0105165e8ad511dc169d5d2068ff07528b3"}
Nov 02 15:08:43.694133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.694103    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2" event={"ID":"2bff10ba-80e4-4be1-873a-53759ead5447","Type":"ContainerStarted","Data":"1849ac7961a4bf0aaad9b575c04c5e3ca7726e7fafd2023bafe2118a685d1108"}
Nov 02 15:08:43.694245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.694150    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2" event={"ID":"2bff10ba-80e4-4be1-873a-53759ead5447","Type":"ContainerStarted","Data":"39eab579df11c7c5293c1411f4080a36ddffd4370746bb25f7e34bca63b9b0a8"}
Nov 02 15:08:43.705830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.705800    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.705924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.705830    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthrk\" (UniqueName: \"kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.705924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.705856    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.739387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.735319    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb" podStartSLOduration=2.735299725 podStartE2EDuration="2.735299725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:43.732787885 +0000 UTC m=+775.990857338" watchObservedRunningTime="2026-11-02 15:08:43.735299725 +0000 UTC m=+775.993369173"
Nov 02 15:08:43.773441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.773386    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2" podStartSLOduration=3.773358393 podStartE2EDuration="3.773358393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:43.770867502 +0000 UTC m=+776.028936951" watchObservedRunningTime="2026-11-02 15:08:43.773358393 +0000 UTC m=+776.031427842"
Nov 02 15:08:43.807333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.807301    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.807333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.807334    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthrk\" (UniqueName: \"kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.807522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.807383    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.808518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.808493    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.810147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.810126    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.819117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.819091    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:08:43.819233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.819141    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:08:43.848806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.848781    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthrk\" (UniqueName: \"kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk\") pod \"maas-api-revoke-sub-b56368a0-ghbv7\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:43.891524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:43.891501    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:44.244309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.243921    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"]
Nov 02 15:08:44.245664 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:44.245635    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f332039_b9c4_4dfe_8f76_ce03ef383c52.slice/crio-dacc9b44adf43d06ce5d85de4094b6741dfcf41e698c06442873041b9f220a0e WatchSource:0}: Error finding container dacc9b44adf43d06ce5d85de4094b6741dfcf41e698c06442873041b9f220a0e: Status 404 returned error can't find the container with id dacc9b44adf43d06ce5d85de4094b6741dfcf41e698c06442873041b9f220a0e
Nov 02 15:08:44.702659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.702622    5041 generic.go:356] "Generic (PLEG): container finished" podID="2bff10ba-80e4-4be1-873a-53759ead5447" containerID="1849ac7961a4bf0aaad9b575c04c5e3ca7726e7fafd2023bafe2118a685d1108" exitCode=0
Nov 02 15:08:44.703047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.702711    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2" event={"ID":"2bff10ba-80e4-4be1-873a-53759ead5447","Type":"ContainerDied","Data":"1849ac7961a4bf0aaad9b575c04c5e3ca7726e7fafd2023bafe2118a685d1108"}
Nov 02 15:08:44.704432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.704409    5041 generic.go:356] "Generic (PLEG): container finished" podID="55e4c139-3746-4611-a926-e1ec04ac43a4" containerID="c6958f09e551b3edd1995fec0c81aa14b18098af75deb455da5f6fbe55e4f7f6" exitCode=0
Nov 02 15:08:44.704536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.704491    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb" event={"ID":"55e4c139-3746-4611-a926-e1ec04ac43a4","Type":"ContainerDied","Data":"c6958f09e551b3edd1995fec0c81aa14b18098af75deb455da5f6fbe55e4f7f6"}
Nov 02 15:08:44.705923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.705905    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7" event={"ID":"1f332039-b9c4-4dfe-8f76-ce03ef383c52","Type":"ContainerStarted","Data":"5e04164fc651633337570809bf8e7b4e263b843e19e2ff7041dcf886b8858afe"}
Nov 02 15:08:44.705976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.705940    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7" event={"ID":"1f332039-b9c4-4dfe-8f76-ce03ef383c52","Type":"ContainerStarted","Data":"dacc9b44adf43d06ce5d85de4094b6741dfcf41e698c06442873041b9f220a0e"}
Nov 02 15:08:44.804844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.804792    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7" podStartSLOduration=1.804779415 podStartE2EDuration="1.804779415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:44.801745712 +0000 UTC m=+777.059815163" watchObservedRunningTime="2026-11-02 15:08:44.804779415 +0000 UTC m=+777.062848864"
Nov 02 15:08:44.825536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.818613    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:08:44.825536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:44.818669    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:08:45.717356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:45.716474    5041 generic.go:356] "Generic (PLEG): container finished" podID="1f332039-b9c4-4dfe-8f76-ce03ef383c52" containerID="5e04164fc651633337570809bf8e7b4e263b843e19e2ff7041dcf886b8858afe" exitCode=0
Nov 02 15:08:45.717356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:45.717135    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7" event={"ID":"1f332039-b9c4-4dfe-8f76-ce03ef383c52","Type":"ContainerDied","Data":"5e04164fc651633337570809bf8e7b4e263b843e19e2ff7041dcf886b8858afe"}
Nov 02 15:08:45.824329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:45.823895    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:08:45.824329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:45.823960    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:08:45.921556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:45.921527    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:45.966175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:45.966146    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:46.060380 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060292    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca\") pod \"55e4c139-3746-4611-a926-e1ec04ac43a4\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") "
Nov 02 15:08:46.060380 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060336    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca\") pod \"55e4c139-3746-4611-a926-e1ec04ac43a4\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") "
Nov 02 15:08:46.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060396    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw\" (UniqueName: \"kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw\") pod \"55e4c139-3746-4611-a926-e1ec04ac43a4\" (UID: \"55e4c139-3746-4611-a926-e1ec04ac43a4\") "
Nov 02 15:08:46.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060448    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca\") pod \"2bff10ba-80e4-4be1-873a-53759ead5447\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") "
Nov 02 15:08:46.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060508    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd\" (UniqueName: \"kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd\") pod \"2bff10ba-80e4-4be1-873a-53759ead5447\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") "
Nov 02 15:08:46.060774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060628    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca\") pod \"2bff10ba-80e4-4be1-873a-53759ead5447\" (UID: \"2bff10ba-80e4-4be1-873a-53759ead5447\") "
Nov 02 15:08:46.060830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.060798    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca" pod "55e4c139-3746-4611-a926-e1ec04ac43a4" (UID: "55e4c139-3746-4611-a926-e1ec04ac43a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:46.061165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.061140    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.061333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.061300    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca" pod "2bff10ba-80e4-4be1-873a-53759ead5447" (UID: "2bff10ba-80e4-4be1-873a-53759ead5447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:46.062684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.062661    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca" pod "55e4c139-3746-4611-a926-e1ec04ac43a4" (UID: "55e4c139-3746-4611-a926-e1ec04ac43a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:46.062791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.062704    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw" pod "55e4c139-3746-4611-a926-e1ec04ac43a4" (UID: "55e4c139-3746-4611-a926-e1ec04ac43a4"). InnerVolumeSpecName "kube-api-access-glxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:46.063081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.063055    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd" pod "2bff10ba-80e4-4be1-873a-53759ead5447" (UID: "2bff10ba-80e4-4be1-873a-53759ead5447"). InnerVolumeSpecName "kube-api-access-mg9rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:46.063081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.063072    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca" pod "2bff10ba-80e4-4be1-873a-53759ead5447" (UID: "2bff10ba-80e4-4be1-873a-53759ead5447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:46.162702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.162666    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e4c139-3746-4611-a926-e1ec04ac43a4-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.162702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.162698    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glxjw\" (UniqueName: \"kubernetes.io/projected/55e4c139-3746-4611-a926-e1ec04ac43a4-kube-api-access-glxjw\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.162942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.162709    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.162942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.162720    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9rd\" (UniqueName: \"kubernetes.io/projected/2bff10ba-80e4-4be1-873a-53759ead5447-kube-api-access-mg9rd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.162942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.162729    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bff10ba-80e4-4be1-873a-53759ead5447-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.724269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.724241    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"
Nov 02 15:08:46.724776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.724270    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2" event={"ID":"2bff10ba-80e4-4be1-873a-53759ead5447","Type":"ContainerDied","Data":"39eab579df11c7c5293c1411f4080a36ddffd4370746bb25f7e34bca63b9b0a8"}
Nov 02 15:08:46.724776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.724347    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39eab579df11c7c5293c1411f4080a36ddffd4370746bb25f7e34bca63b9b0a8"
Nov 02 15:08:46.725701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.725681    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"
Nov 02 15:08:46.725816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.725727    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb" event={"ID":"55e4c139-3746-4611-a926-e1ec04ac43a4","Type":"ContainerDied","Data":"533f67f5a126abc24eadd91c7a84f0105165e8ad511dc169d5d2068ff07528b3"}
Nov 02 15:08:46.725816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.725769    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533f67f5a126abc24eadd91c7a84f0105165e8ad511dc169d5d2068ff07528b3"
Nov 02 15:08:46.822272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.822237    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:46.822606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.822583    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:46.823431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.823408    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:08:46.878224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.878145    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:46.974604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.974524    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca\") pod \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") "
Nov 02 15:08:46.974738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.974709    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca\") pod \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") "
Nov 02 15:08:46.974799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.974772    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk\" (UniqueName: \"kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk\") pod \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\" (UID: \"1f332039-b9c4-4dfe-8f76-ce03ef383c52\") "
Nov 02 15:08:46.974987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.974950    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca" pod "1f332039-b9c4-4dfe-8f76-ce03ef383c52" (UID: "1f332039-b9c4-4dfe-8f76-ce03ef383c52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:46.975639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.975610    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:46.977889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.977859    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca" pod "1f332039-b9c4-4dfe-8f76-ce03ef383c52" (UID: "1f332039-b9c4-4dfe-8f76-ce03ef383c52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:46.983503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:46.983480    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk" pod "1f332039-b9c4-4dfe-8f76-ce03ef383c52" (UID: "1f332039-b9c4-4dfe-8f76-ce03ef383c52"). InnerVolumeSpecName "kube-api-access-fthrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:47.076493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:47.076457    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f332039-b9c4-4dfe-8f76-ce03ef383c52-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:47.076493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:47.076487    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthrk\" (UniqueName: \"kubernetes.io/projected/1f332039-b9c4-4dfe-8f76-ce03ef383c52-kube-api-access-fthrk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:47.733918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:47.733886    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"
Nov 02 15:08:47.734457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:47.733923    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7" event={"ID":"1f332039-b9c4-4dfe-8f76-ce03ef383c52","Type":"ContainerDied","Data":"dacc9b44adf43d06ce5d85de4094b6741dfcf41e698c06442873041b9f220a0e"}
Nov 02 15:08:47.734457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:47.733975    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dacc9b44adf43d06ce5d85de4094b6741dfcf41e698c06442873041b9f220a0e"
Nov 02 15:08:49.600750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.600701    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"]
Nov 02 15:08:49.602681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602651    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1f332039-b9c4-4dfe-8f76-ce03ef383c52" containerName="revoke-keys"
Nov 02 15:08:49.602681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602680    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f332039-b9c4-4dfe-8f76-ce03ef383c52" containerName="revoke-keys"
Nov 02 15:08:49.602872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602734    5041 cpu_manager.go:414] "containerMap: removing container" podUID="55e4c139-3746-4611-a926-e1ec04ac43a4" containerName="revoke-keys"
Nov 02 15:08:49.602872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602744    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e4c139-3746-4611-a926-e1ec04ac43a4" containerName="revoke-keys"
Nov 02 15:08:49.602872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602789    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2bff10ba-80e4-4be1-873a-53759ead5447" containerName="revoke-keys"
Nov 02 15:08:49.602872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602798    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bff10ba-80e4-4be1-873a-53759ead5447" containerName="revoke-keys"
Nov 02 15:08:49.603087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602949    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f332039-b9c4-4dfe-8f76-ce03ef383c52" containerName="revoke-keys"
Nov 02 15:08:49.603087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602965    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e4c139-3746-4611-a926-e1ec04ac43a4" containerName="revoke-keys"
Nov 02 15:08:49.603087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.602980    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bff10ba-80e4-4be1-873a-53759ead5447" containerName="revoke-keys"
Nov 02 15:08:49.606619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.606594    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:49.631803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.631767    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:08:49.632794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.632774    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:08:49.634063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.634042    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"]
Nov 02 15:08:49.920215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.917706    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:49.920215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.917803    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:49.920215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:49.917859    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnzs\" (UniqueName: \"kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.039688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.039625    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.041819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.041789    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.042041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.042022    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnzs\" (UniqueName: \"kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.042555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.041394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.047828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.047805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.099386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.097128    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnzs\" (UniqueName: \"kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs\") pod \"maas-api-revoke-sub-6570bcb8-mr7cm\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.171624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.171530    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:50.542590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.539443    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"]
Nov 02 15:08:50.624690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.624653    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"]
Nov 02 15:08:50.625149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.624827    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.737388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.702457    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.737388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.702562    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrn87\" (UniqueName: \"kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.737388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.702645    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.737388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.712431    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"]
Nov 02 15:08:50.803439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.803406    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrn87\" (UniqueName: \"kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.803576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.803490    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.803576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.803573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.804311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.804287    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.806177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.806144    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.876397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.871225    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrn87\" (UniqueName: \"kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87\") pod \"maas-api-revoke-sub-ba8f6ec2-nr45f\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:50.906786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:50.906749    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm" event={"ID":"bdbed397-26d7-40c8-9b41-b3a5e47da1a2","Type":"ContainerStarted","Data":"cdda7ddd50f9adff3d532e31cef23b331a15f27527e29bffa48abf19788177be"}
Nov 02 15:08:51.086391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.086288    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:51.231015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.226585    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"]
Nov 02 15:08:51.234830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.232542    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.256713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.254054    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"]
Nov 02 15:08:51.308671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.308633    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.308911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.308887    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.309195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.309174    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58296\" (UniqueName: \"kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.410129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.410033    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58296\" (UniqueName: \"kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.410129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.410121    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.410400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.410157    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.410768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.410742    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.412622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.412592    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.433409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.433343    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58296\" (UniqueName: \"kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296\") pod \"maas-api-revoke-sub-02d0ec2a-dqpnj\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.530234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.530210    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"]
Nov 02 15:08:51.530933 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:51.530910    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712fd78e_59cc_4719_bc0b_534045981ec2.slice/crio-9ae3a0e40fe1727dde05901f982e364eb9bebf64d49266fcf5b2a52b1aecfc9a WatchSource:0}: Error finding container 9ae3a0e40fe1727dde05901f982e364eb9bebf64d49266fcf5b2a52b1aecfc9a: Status 404 returned error can't find the container with id 9ae3a0e40fe1727dde05901f982e364eb9bebf64d49266fcf5b2a52b1aecfc9a
Nov 02 15:08:51.608892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.608849    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:51.917749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.917716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm" event={"ID":"bdbed397-26d7-40c8-9b41-b3a5e47da1a2","Type":"ContainerStarted","Data":"1f96fa398e791c7c7573f130915a5add48b109879417a01c5aa4051aa8727750"}
Nov 02 15:08:51.920553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.920521    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f" event={"ID":"712fd78e-59cc-4719-bc0b-534045981ec2","Type":"ContainerStarted","Data":"9ae3a0e40fe1727dde05901f982e364eb9bebf64d49266fcf5b2a52b1aecfc9a"}
Nov 02 15:08:51.962180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:51.962117    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm" podStartSLOduration=2.962096978 podStartE2EDuration="2.962096978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:51.958125031 +0000 UTC m=+784.216194515" watchObservedRunningTime="2026-11-02 15:08:51.962096978 +0000 UTC m=+784.220166429"
Nov 02 15:08:52.006894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.006868    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"]
Nov 02 15:08:52.008237 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:52.008215    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec55a09e_8298_4e6a_b3c5_20f7e777e2be.slice/crio-afdc51566918a8dd818a2705e0c303f7ca3312415e0f26b730c3b3c2feb635b0 WatchSource:0}: Error finding container afdc51566918a8dd818a2705e0c303f7ca3312415e0f26b730c3b3c2feb635b0: Status 404 returned error can't find the container with id afdc51566918a8dd818a2705e0c303f7ca3312415e0f26b730c3b3c2feb635b0
Nov 02 15:08:52.928290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.928253    5041 generic.go:356] "Generic (PLEG): container finished" podID="ec55a09e-8298-4e6a-b3c5-20f7e777e2be" containerID="d8c80bd88e63fb9c32b8dd9e978c597f47d3ae09b8e4d7cd5cb1cc522ae1744d" exitCode=0
Nov 02 15:08:52.928702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.928335    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj" event={"ID":"ec55a09e-8298-4e6a-b3c5-20f7e777e2be","Type":"ContainerDied","Data":"d8c80bd88e63fb9c32b8dd9e978c597f47d3ae09b8e4d7cd5cb1cc522ae1744d"}
Nov 02 15:08:52.928702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.928410    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj" event={"ID":"ec55a09e-8298-4e6a-b3c5-20f7e777e2be","Type":"ContainerStarted","Data":"afdc51566918a8dd818a2705e0c303f7ca3312415e0f26b730c3b3c2feb635b0"}
Nov 02 15:08:52.930044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.930021    5041 generic.go:356] "Generic (PLEG): container finished" podID="bdbed397-26d7-40c8-9b41-b3a5e47da1a2" containerID="1f96fa398e791c7c7573f130915a5add48b109879417a01c5aa4051aa8727750" exitCode=0
Nov 02 15:08:52.930143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.930093    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm" event={"ID":"bdbed397-26d7-40c8-9b41-b3a5e47da1a2","Type":"ContainerDied","Data":"1f96fa398e791c7c7573f130915a5add48b109879417a01c5aa4051aa8727750"}
Nov 02 15:08:52.931555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.931534    5041 generic.go:356] "Generic (PLEG): container finished" podID="712fd78e-59cc-4719-bc0b-534045981ec2" containerID="c0533ce0dd59feca9be265d0d35439d8bc9032b2341bd9db0cb9b85f460c0d77" exitCode=0
Nov 02 15:08:52.931670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:52.931612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f" event={"ID":"712fd78e-59cc-4719-bc0b-534045981ec2","Type":"ContainerDied","Data":"c0533ce0dd59feca9be265d0d35439d8bc9032b2341bd9db0cb9b85f460c0d77"}
Nov 02 15:08:54.114343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.114305    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:54.165225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.165197    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:54.169236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.169213    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:54.174691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.174664    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"]
Nov 02 15:08:54.175277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175258    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bdbed397-26d7-40c8-9b41-b3a5e47da1a2" containerName="revoke-keys"
Nov 02 15:08:54.175277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175276    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbed397-26d7-40c8-9b41-b3a5e47da1a2" containerName="revoke-keys"
Nov 02 15:08:54.175441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175289    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ec55a09e-8298-4e6a-b3c5-20f7e777e2be" containerName="revoke-keys"
Nov 02 15:08:54.175441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175294    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec55a09e-8298-4e6a-b3c5-20f7e777e2be" containerName="revoke-keys"
Nov 02 15:08:54.175441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175319    5041 cpu_manager.go:414] "containerMap: removing container" podUID="712fd78e-59cc-4719-bc0b-534045981ec2" containerName="revoke-keys"
Nov 02 15:08:54.175441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175324    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712fd78e-59cc-4719-bc0b-534045981ec2" containerName="revoke-keys"
Nov 02 15:08:54.175620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175458    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="712fd78e-59cc-4719-bc0b-534045981ec2" containerName="revoke-keys"
Nov 02 15:08:54.175620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175470    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbed397-26d7-40c8-9b41-b3a5e47da1a2" containerName="revoke-keys"
Nov 02 15:08:54.175620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.175475    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec55a09e-8298-4e6a-b3c5-20f7e777e2be" containerName="revoke-keys"
Nov 02 15:08:54.182506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.182481    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.196679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.196621    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6af50f-openshift-default-dockercfg-n5jqm\""
Nov 02 15:08:54.197157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.197131    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"]
Nov 02 15:08:54.243282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.243254    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca\") pod \"712fd78e-59cc-4719-bc0b-534045981ec2\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") "
Nov 02 15:08:54.243440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.243416    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca\") pod \"712fd78e-59cc-4719-bc0b-534045981ec2\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") "
Nov 02 15:08:54.243493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.243450    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87\" (UniqueName: \"kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87\") pod \"712fd78e-59cc-4719-bc0b-534045981ec2\" (UID: \"712fd78e-59cc-4719-bc0b-534045981ec2\") "
Nov 02 15:08:54.243880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.243843    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca" pod "712fd78e-59cc-4719-bc0b-534045981ec2" (UID: "712fd78e-59cc-4719-bc0b-534045981ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:54.246096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.246066    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87" pod "712fd78e-59cc-4719-bc0b-534045981ec2" (UID: "712fd78e-59cc-4719-bc0b-534045981ec2"). InnerVolumeSpecName "kube-api-access-lrn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:54.246179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.246129    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca" pod "712fd78e-59cc-4719-bc0b-534045981ec2" (UID: "712fd78e-59cc-4719-bc0b-534045981ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:54.345094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345057    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296\" (UniqueName: \"kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296\") pod \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") "
Nov 02 15:08:54.345251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345117    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca\") pod \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") "
Nov 02 15:08:54.345251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345150    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca\") pod \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") "
Nov 02 15:08:54.345251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345190    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs\" (UniqueName: \"kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs\") pod \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") "
Nov 02 15:08:54.345251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345218    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca\") pod \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\" (UID: \"bdbed397-26d7-40c8-9b41-b3a5e47da1a2\") "
Nov 02 15:08:54.345614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345544    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca\") pod \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\" (UID: \"ec55a09e-8298-4e6a-b3c5-20f7e777e2be\") "
Nov 02 15:08:54.346134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345694    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca" pod "ec55a09e-8298-4e6a-b3c5-20f7e777e2be" (UID: "ec55a09e-8298-4e6a-b3c5-20f7e777e2be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:54.346134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345796    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345974    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.345980    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca" pod "bdbed397-26d7-40c8-9b41-b3a5e47da1a2" (UID: "bdbed397-26d7-40c8-9b41-b3a5e47da1a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:08:54.346134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346025    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346056    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346190    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346253    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346295    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346422    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwpd\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346528    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.346810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346593    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.346810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346613    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.346810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346628    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712fd78e-59cc-4719-bc0b-534045981ec2-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.346810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346664    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrn87\" (UniqueName: \"kubernetes.io/projected/712fd78e-59cc-4719-bc0b-534045981ec2-kube-api-access-lrn87\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.346810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.346684    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.347509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.347485    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296" pod "ec55a09e-8298-4e6a-b3c5-20f7e777e2be" (UID: "ec55a09e-8298-4e6a-b3c5-20f7e777e2be"). InnerVolumeSpecName "kube-api-access-58296". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:54.347811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.347789    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca" pod "bdbed397-26d7-40c8-9b41-b3a5e47da1a2" (UID: "bdbed397-26d7-40c8-9b41-b3a5e47da1a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:54.347878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.347794    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca" pod "ec55a09e-8298-4e6a-b3c5-20f7e777e2be" (UID: "ec55a09e-8298-4e6a-b3c5-20f7e777e2be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:08:54.347938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.347923    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs" pod "bdbed397-26d7-40c8-9b41-b3a5e47da1a2" (UID: "bdbed397-26d7-40c8-9b41-b3a5e47da1a2"). InnerVolumeSpecName "kube-api-access-rsnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:08:54.447739 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447642    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.447739 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447734    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.447952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447766    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.447952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447795    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.447952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447842    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.447952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447888    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.447952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447923    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.447967    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwpd\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448161    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448177    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448359    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58296\" (UniqueName: \"kubernetes.io/projected/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-kube-api-access-58296\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.448404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448398    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.448491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448412    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448416    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsnzs\" (UniqueName: \"kubernetes.io/projected/bdbed397-26d7-40c8-9b41-b3a5e47da1a2-kube-api-access-rsnzs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.448491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448466    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55a09e-8298-4e6a-b3c5-20f7e777e2be-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:08:54.448491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448444    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.448959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.448935    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.450036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.450017    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.450307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.450286    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.470436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.470416    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.470592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.470575    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwpd\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.498619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.498595    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:54.815867 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:08:54.815835    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823c93a5_c32c_413e_ae47_392e56b40af4.slice/crio-35d61e9fdd301baef01a700abf698e4b68fc4bb1adbd42e4d4c64a611e73d610 WatchSource:0}: Error finding container 35d61e9fdd301baef01a700abf698e4b68fc4bb1adbd42e4d4c64a611e73d610: Status 404 returned error can't find the container with id 35d61e9fdd301baef01a700abf698e4b68fc4bb1adbd42e4d4c64a611e73d610
Nov 02 15:08:54.816384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.816339    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"]
Nov 02 15:08:54.818488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.818456    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:54.818557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.818519    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:54.818557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.818549    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:54.950586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.950555    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"
Nov 02 15:08:54.950749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.950556    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj" event={"ID":"ec55a09e-8298-4e6a-b3c5-20f7e777e2be","Type":"ContainerDied","Data":"afdc51566918a8dd818a2705e0c303f7ca3312415e0f26b730c3b3c2feb635b0"}
Nov 02 15:08:54.950749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.950694    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afdc51566918a8dd818a2705e0c303f7ca3312415e0f26b730c3b3c2feb635b0"
Nov 02 15:08:54.952148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.952119    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"
Nov 02 15:08:54.952264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.952117    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm" event={"ID":"bdbed397-26d7-40c8-9b41-b3a5e47da1a2","Type":"ContainerDied","Data":"cdda7ddd50f9adff3d532e31cef23b331a15f27527e29bffa48abf19788177be"}
Nov 02 15:08:54.952264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.952243    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdda7ddd50f9adff3d532e31cef23b331a15f27527e29bffa48abf19788177be"
Nov 02 15:08:54.959845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.954043    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"
Nov 02 15:08:54.959845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.954046    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f" event={"ID":"712fd78e-59cc-4719-bc0b-534045981ec2","Type":"ContainerDied","Data":"9ae3a0e40fe1727dde05901f982e364eb9bebf64d49266fcf5b2a52b1aecfc9a"}
Nov 02 15:08:54.959845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.954094    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae3a0e40fe1727dde05901f982e364eb9bebf64d49266fcf5b2a52b1aecfc9a"
Nov 02 15:08:54.959845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.959800    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" event={"ID":"823c93a5-c32c-413e-ae47-392e56b40af4","Type":"ContainerStarted","Data":"220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384"}
Nov 02 15:08:54.959845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:54.959845    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" event={"ID":"823c93a5-c32c-413e-ae47-392e56b40af4","Type":"ContainerStarted","Data":"35d61e9fdd301baef01a700abf698e4b68fc4bb1adbd42e4d4c64a611e73d610"}
Nov 02 15:08:55.013837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:55.013780    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podStartSLOduration=1.013765411 podStartE2EDuration="1.013765411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:55.003093224 +0000 UTC m=+787.261162675" watchObservedRunningTime="2026-11-02 15:08:55.013765411 +0000 UTC m=+787.271834861"
Nov 02 15:08:55.499446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:55.499402    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:08:55.501446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:55.501416    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 15:08:55.501576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:55.501476    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 15:08:56.499786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.499751    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 15:08:56.500185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.499803    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 15:08:56.872661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.872581    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"]
Nov 02 15:08:56.875947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.875930    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:56.890120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.890092    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:08:56.901596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.901569    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:08:56.908176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.908135    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"]
Nov 02 15:08:56.975796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.975764    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjdd\" (UniqueName: \"kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:56.975999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.975891    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:56.975999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:56.975954    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.077337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.077295    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjdd\" (UniqueName: \"kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.077536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.077478    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.077590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.077573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.078141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.078122    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.080013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.079992    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.100308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.100283    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjdd\" (UniqueName: \"kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd\") pod \"maas-api-revoke-sub-15d60855-x5z2z\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.189043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.189011    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:08:57.510834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.510237    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 15:08:57.510834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.510304    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 15:08:57.551177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.551069    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"]
Nov 02 15:08:57.984060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:57.984022    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z" event={"ID":"9a78d457-513d-4016-a908-878b6d0c9f54","Type":"ContainerStarted","Data":"067fbbec287c87e6e662142b7ac63aae1beee985563830b1501501746b7f8ef7"}
Nov 02 15:08:58.499550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:58.499515    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 15:08:58.499768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:58.499567    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 15:08:58.993173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:58.993143    5041 generic.go:356] "Generic (PLEG): container finished" podID="9a78d457-513d-4016-a908-878b6d0c9f54" containerID="38cfff588cedc0f5b6f9cafcc4342366f761618f8c9a86adfc364694ae50ab79" exitCode=0
Nov 02 15:08:58.993568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:58.993226    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z" event={"ID":"9a78d457-513d-4016-a908-878b6d0c9f54","Type":"ContainerDied","Data":"38cfff588cedc0f5b6f9cafcc4342366f761618f8c9a86adfc364694ae50ab79"}
Nov 02 15:08:59.501915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:59.501743    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 15:08:59.501915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:08:59.501804    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 15:09:00.184943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.184919    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:09:00.259693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.259589    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca\") pod \"9a78d457-513d-4016-a908-878b6d0c9f54\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") "
Nov 02 15:09:00.259693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.259645    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca\") pod \"9a78d457-513d-4016-a908-878b6d0c9f54\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") "
Nov 02 15:09:00.259980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.259812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd\" (UniqueName: \"kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd\") pod \"9a78d457-513d-4016-a908-878b6d0c9f54\" (UID: \"9a78d457-513d-4016-a908-878b6d0c9f54\") "
Nov 02 15:09:00.260065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.260036    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca" pod "9a78d457-513d-4016-a908-878b6d0c9f54" (UID: "9a78d457-513d-4016-a908-878b6d0c9f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:09:00.262129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.262105    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd" pod "9a78d457-513d-4016-a908-878b6d0c9f54" (UID: "9a78d457-513d-4016-a908-878b6d0c9f54"). InnerVolumeSpecName "kube-api-access-jrjdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:09:00.262471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.262441    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca" pod "9a78d457-513d-4016-a908-878b6d0c9f54" (UID: "9a78d457-513d-4016-a908-878b6d0c9f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:09:00.361353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.361316    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:00.361353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.361347    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjdd\" (UniqueName: \"kubernetes.io/projected/9a78d457-513d-4016-a908-878b6d0c9f54-kube-api-access-jrjdd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:00.361353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.361357    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a78d457-513d-4016-a908-878b6d0c9f54-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:00.499755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.499722    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 15:09:00.499922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:00.499775    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 15:09:01.017828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:01.017794    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"
Nov 02 15:09:01.018038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:01.017826    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z" event={"ID":"9a78d457-513d-4016-a908-878b6d0c9f54","Type":"ContainerDied","Data":"067fbbec287c87e6e662142b7ac63aae1beee985563830b1501501746b7f8ef7"}
Nov 02 15:09:01.018038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:01.017898    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="067fbbec287c87e6e662142b7ac63aae1beee985563830b1501501746b7f8ef7"
Nov 02 15:09:01.741741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:01.741705    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:09:01.742243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:01.742113    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:09:01.742939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:01.742919    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:09:03.975765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:03.975729    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"]
Nov 02 15:09:03.976601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:03.976567    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9a78d457-513d-4016-a908-878b6d0c9f54" containerName="revoke-keys"
Nov 02 15:09:03.976601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:03.976602    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a78d457-513d-4016-a908-878b6d0c9f54" containerName="revoke-keys"
Nov 02 15:09:03.976774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:03.976730    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a78d457-513d-4016-a908-878b6d0c9f54" containerName="revoke-keys"
Nov 02 15:09:03.979207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:03.979186    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:03.993489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:03.993469    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:09:04.000087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.000064    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48h9p\" (UniqueName: \"kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.000226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.000199    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.000316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.000298    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.005132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.005108    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:09:04.011761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.011733    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"]
Nov 02 15:09:04.101937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.101905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.102102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.101958    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.102102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.102072    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48h9p\" (UniqueName: \"kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.102642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.102622    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.104259 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.104243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.124726 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.124696    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48h9p\" (UniqueName: \"kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p\") pod \"maas-api-revoke-sub-8b89b69b-r6l6r\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.291296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.291208    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:04.594414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:04.594357    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"]
Nov 02 15:09:04.594636 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:04.594612    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862edf1c_a264_42ed_9746_f7b2261e3dee.slice/crio-69093e2789955907a81d8835331663ac6891686451ad793465f3816eb98e9ec9 WatchSource:0}: Error finding container 69093e2789955907a81d8835331663ac6891686451ad793465f3816eb98e9ec9: Status 404 returned error can't find the container with id 69093e2789955907a81d8835331663ac6891686451ad793465f3816eb98e9ec9
Nov 02 15:09:05.049319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:05.049267    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r" event={"ID":"862edf1c-a264-42ed-9746-f7b2261e3dee","Type":"ContainerStarted","Data":"6d74da1eff39cc1c4ef1d1f72a4c86679c3e03279b3fd7810710149a04cae0f2"}
Nov 02 15:09:05.049745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:05.049381    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r" event={"ID":"862edf1c-a264-42ed-9746-f7b2261e3dee","Type":"ContainerStarted","Data":"69093e2789955907a81d8835331663ac6891686451ad793465f3816eb98e9ec9"}
Nov 02 15:09:05.099079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:05.099028    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r" podStartSLOduration=2.099014031 podStartE2EDuration="2.099014031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:05.094665349 +0000 UTC m=+797.352734801" watchObservedRunningTime="2026-11-02 15:09:05.099014031 +0000 UTC m=+797.357083480"
Nov 02 15:09:06.058291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:06.058255    5041 generic.go:356] "Generic (PLEG): container finished" podID="862edf1c-a264-42ed-9746-f7b2261e3dee" containerID="6d74da1eff39cc1c4ef1d1f72a4c86679c3e03279b3fd7810710149a04cae0f2" exitCode=0
Nov 02 15:09:06.058702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:06.058340    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r" event={"ID":"862edf1c-a264-42ed-9746-f7b2261e3dee","Type":"ContainerDied","Data":"6d74da1eff39cc1c4ef1d1f72a4c86679c3e03279b3fd7810710149a04cae0f2"}
Nov 02 15:09:06.769028 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:09:06.768988    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:09:07.223716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.223690    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:07.308812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.308775    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"]
Nov 02 15:09:07.309465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.309442    5041 cpu_manager.go:414] "containerMap: removing container" podUID="862edf1c-a264-42ed-9746-f7b2261e3dee" containerName="revoke-keys"
Nov 02 15:09:07.309465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.309461    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862edf1c-a264-42ed-9746-f7b2261e3dee" containerName="revoke-keys"
Nov 02 15:09:07.309676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.309580    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="862edf1c-a264-42ed-9746-f7b2261e3dee" containerName="revoke-keys"
Nov 02 15:09:07.312080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.312062    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.336447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.336106    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"]
Nov 02 15:09:07.343649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.343618    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca\") pod \"862edf1c-a264-42ed-9746-f7b2261e3dee\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") "
Nov 02 15:09:07.343816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.343673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca\") pod \"862edf1c-a264-42ed-9746-f7b2261e3dee\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") "
Nov 02 15:09:07.343816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.343707    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p\" (UniqueName: \"kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p\") pod \"862edf1c-a264-42ed-9746-f7b2261e3dee\" (UID: \"862edf1c-a264-42ed-9746-f7b2261e3dee\") "
Nov 02 15:09:07.343987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.343963    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.344042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.343994    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca" pod "862edf1c-a264-42ed-9746-f7b2261e3dee" (UID: "862edf1c-a264-42ed-9746-f7b2261e3dee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:09:07.344042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.344012    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.344191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.344176    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx25j\" (UniqueName: \"kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.344268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.344256    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:07.346597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.346575    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca" pod "862edf1c-a264-42ed-9746-f7b2261e3dee" (UID: "862edf1c-a264-42ed-9746-f7b2261e3dee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:09:07.346720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.346610    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p" pod "862edf1c-a264-42ed-9746-f7b2261e3dee" (UID: "862edf1c-a264-42ed-9746-f7b2261e3dee"). InnerVolumeSpecName "kube-api-access-48h9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:09:07.445057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.445023    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx25j\" (UniqueName: \"kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.445238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.445102    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.445238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.445124    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.445238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.445193    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862edf1c-a264-42ed-9746-f7b2261e3dee-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:07.445238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.445206    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48h9p\" (UniqueName: \"kubernetes.io/projected/862edf1c-a264-42ed-9746-f7b2261e3dee-kube-api-access-48h9p\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:07.445840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.445805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.447590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.447571    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.468041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.468008    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx25j\" (UniqueName: \"kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j\") pod \"maas-api-revoke-sub-e2b88472-pfqm4\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.628347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.628267    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:07.933856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:07.933828    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"]
Nov 02 15:09:07.934623 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:07.934592    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8761c5e7_df26_4f7b_9526_675d8c9995e3.slice/crio-18b9c9473390e2518796e4e107ce8d8bfbda2d208fe69e8e9c6db470a1b9abee WatchSource:0}: Error finding container 18b9c9473390e2518796e4e107ce8d8bfbda2d208fe69e8e9c6db470a1b9abee: Status 404 returned error can't find the container with id 18b9c9473390e2518796e4e107ce8d8bfbda2d208fe69e8e9c6db470a1b9abee
Nov 02 15:09:08.080744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.080714    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"
Nov 02 15:09:08.080744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.080735    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r" event={"ID":"862edf1c-a264-42ed-9746-f7b2261e3dee","Type":"ContainerDied","Data":"69093e2789955907a81d8835331663ac6891686451ad793465f3816eb98e9ec9"}
Nov 02 15:09:08.081002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.080806    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69093e2789955907a81d8835331663ac6891686451ad793465f3816eb98e9ec9"
Nov 02 15:09:08.081953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.081929    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4" event={"ID":"8761c5e7-df26-4f7b-9526-675d8c9995e3","Type":"ContainerStarted","Data":"18b9c9473390e2518796e4e107ce8d8bfbda2d208fe69e8e9c6db470a1b9abee"}
Nov 02 15:09:08.685224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.685188    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"]
Nov 02 15:09:08.688721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.688699    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.702956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.702932    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1cc1b2-openshift-default-dockercfg-vsxtg\""
Nov 02 15:09:08.719744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.719716    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"]
Nov 02 15:09:08.767381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767326    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767455    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767540    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767672    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767820    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767852    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.767978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.767953    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6v5\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.768043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.768029    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869522    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869582    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869651    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6v5\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869693    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869757    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869791    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869844    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.869878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869871    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.870263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.869917    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.872082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.870985    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.872082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.870998    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.872082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.871232    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.872082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.871575    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.872082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.872020    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.875893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.874937    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.879445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.879419    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"]
Nov 02 15:09:08.883945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.883905    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.888491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.888468    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.901532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.901509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.902214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.902008    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6v5\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:08.902214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.902180    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5dd8e9-openshift-default-dockercfg-fthlq\""
Nov 02 15:09:08.926171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.922532    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"]
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.970687    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.970734    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.970767    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.970980    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.971032    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.971078    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.971110    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.971175    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqxj\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.973455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.971274    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:08.997232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:08.997178    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"]
Nov 02 15:09:09.001938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.001897    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:09.025289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.024350    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.034333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.034306    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"]
Nov 02 15:09:09.040338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.040313    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-739bf1-openshift-default-dockercfg-dm45l\""
Nov 02 15:09:09.072986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.072950    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.073174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073019    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.073174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073084    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.073174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073121    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.073174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073161    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073187    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073235    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073275    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073301    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073332    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073587    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqxj\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073653    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073737    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073765    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh488\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073825    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.073904    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.074050    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.074053    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.074087    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.074185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.074263    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.076452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.074322    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.076452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.075358    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.076755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.076715    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.079452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.079429    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.080449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.080423    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.094405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.094319    5041 generic.go:356] "Generic (PLEG): container finished" podID="8761c5e7-df26-4f7b-9526-675d8c9995e3" containerID="d8d613f4b1eb60ede0ed5f12fa96df2f457e1da1314afcd38eed976cc15a19de" exitCode=0
Nov 02 15:09:09.094531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.094447    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4" event={"ID":"8761c5e7-df26-4f7b-9526-675d8c9995e3","Type":"ContainerDied","Data":"d8d613f4b1eb60ede0ed5f12fa96df2f457e1da1314afcd38eed976cc15a19de"}
Nov 02 15:09:09.097512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.097488    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqxj\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.097650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.097538    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.176674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.176866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176723    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh488\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.176866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176766    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.176866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176812    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.177073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176868    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.177073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176938    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.177073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.176976    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.177073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.177017    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.177299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.177088    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.177666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.177630    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.180923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.180612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.185283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.184208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.185283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.184975    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.185283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.184981    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.185283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.185243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.185283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.185271    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.214443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.213420    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh488\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.235460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.231887    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.304023 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.303975    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:09.372864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.372828    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:09.379467 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.378563    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"]
Nov 02 15:09:09.425191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.424224    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.425191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.424322    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.425191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.424397    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.673650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.656448    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"]
Nov 02 15:09:09.701599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.701522    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.702105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.701632    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.702105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.701681    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.867628 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:09.867581    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b0bd35_d966_45a0_84a2_6d0d492fa1c7.slice/crio-99d89a2c54539e4b73309c4857849c18955f9757dee43b8d2c02097f0adc420e WatchSource:0}: Error finding container 99d89a2c54539e4b73309c4857849c18955f9757dee43b8d2c02097f0adc420e: Status 404 returned error can't find the container with id 99d89a2c54539e4b73309c4857849c18955f9757dee43b8d2c02097f0adc420e
Nov 02 15:09:09.872155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.872116    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.872391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.872348    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.872559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.872535    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.903088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:09.903057    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"]
Nov 02 15:09:10.107869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.107838    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" event={"ID":"cf5eb9e3-d415-402a-b228-71fc41dd80fe","Type":"ContainerStarted","Data":"4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a"}
Nov 02 15:09:10.109571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.109440    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" event={"ID":"cf5eb9e3-d415-402a-b228-71fc41dd80fe","Type":"ContainerStarted","Data":"fbd71136ccb93e5c691c9c14594a625efac21dfcb777193e6b3a7e3ab75bf88b"}
Nov 02 15:09:10.110884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.110407    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" event={"ID":"bdbac19b-fed9-4bcd-a66d-ef9d24babefb","Type":"ContainerStarted","Data":"4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea"}
Nov 02 15:09:10.110884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.110455    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" event={"ID":"bdbac19b-fed9-4bcd-a66d-ef9d24babefb","Type":"ContainerStarted","Data":"43cf3d1d3b02587b03d8afe630c12837b2a1492dc70755a39f859efd3fd29d58"}
Nov 02 15:09:10.113793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.113696    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" event={"ID":"44b0bd35-d966-45a0-84a2-6d0d492fa1c7","Type":"ContainerStarted","Data":"dfaa8d1035d9156b5b76609127280b290a3f52e9e0bb5a1bca3d323002403a05"}
Nov 02 15:09:10.113793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.113740    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" event={"ID":"44b0bd35-d966-45a0-84a2-6d0d492fa1c7","Type":"ContainerStarted","Data":"99d89a2c54539e4b73309c4857849c18955f9757dee43b8d2c02097f0adc420e"}
Nov 02 15:09:10.180253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.180145    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podStartSLOduration=2.180126412 podStartE2EDuration="2.180126412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:10.156673978 +0000 UTC m=+802.414743452" watchObservedRunningTime="2026-11-02 15:09:10.180126412 +0000 UTC m=+802.438195865"
Nov 02 15:09:10.217677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.217618    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podStartSLOduration=2.217597609 podStartE2EDuration="2.217597609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:10.214550093 +0000 UTC m=+802.472619567" watchObservedRunningTime="2026-11-02 15:09:10.217597609 +0000 UTC m=+802.475667063"
Nov 02 15:09:10.284518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.282388    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podStartSLOduration=2.282353524 podStartE2EDuration="2.282353524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:10.276762652 +0000 UTC m=+802.534832190" watchObservedRunningTime="2026-11-02 15:09:10.282353524 +0000 UTC m=+802.540422966"
Nov 02 15:09:10.306491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.306454    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:10.396014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.395872    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:10.482740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.482713    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:10.540777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.540149    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j\" (UniqueName: \"kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j\") pod \"8761c5e7-df26-4f7b-9526-675d8c9995e3\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") "
Nov 02 15:09:10.540777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.540231    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca\") pod \"8761c5e7-df26-4f7b-9526-675d8c9995e3\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") "
Nov 02 15:09:10.540777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.540270    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca\") pod \"8761c5e7-df26-4f7b-9526-675d8c9995e3\" (UID: \"8761c5e7-df26-4f7b-9526-675d8c9995e3\") "
Nov 02 15:09:10.544734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.544702    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca" pod "8761c5e7-df26-4f7b-9526-675d8c9995e3" (UID: "8761c5e7-df26-4f7b-9526-675d8c9995e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:09:10.562271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.561721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j" pod "8761c5e7-df26-4f7b-9526-675d8c9995e3" (UID: "8761c5e7-df26-4f7b-9526-675d8c9995e3"). InnerVolumeSpecName "kube-api-access-nx25j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:09:10.565395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.565109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca" pod "8761c5e7-df26-4f7b-9526-675d8c9995e3" (UID: "8761c5e7-df26-4f7b-9526-675d8c9995e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:09:10.644745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.644696    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:10.645022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.645004    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx25j\" (UniqueName: \"kubernetes.io/projected/8761c5e7-df26-4f7b-9526-675d8c9995e3-kube-api-access-nx25j\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:10.645162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:10.645147    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8761c5e7-df26-4f7b-9526-675d8c9995e3-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:11.002765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.002733    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:11.004883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.004844    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:09:11.004990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.004905    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:09:11.195939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.195883    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4" event={"ID":"8761c5e7-df26-4f7b-9526-675d8c9995e3","Type":"ContainerDied","Data":"18b9c9473390e2518796e4e107ce8d8bfbda2d208fe69e8e9c6db470a1b9abee"}
Nov 02 15:09:11.196314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.196281    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b9c9473390e2518796e4e107ce8d8bfbda2d208fe69e8e9c6db470a1b9abee"
Nov 02 15:09:11.196497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.196457    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"
Nov 02 15:09:11.307124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.307032    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 15:09:11.307124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.307085    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 15:09:11.393413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.385480    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:09:11.393413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:11.385545    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:09:12.003882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:12.003843    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.004308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:12.003922    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:09:12.305082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:12.304988    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.305082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:12.305051    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 15:09:12.376388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:12.375994    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.376388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:12.376052    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:09:13.003429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:13.003352    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.003599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:13.003448    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:09:13.304927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:13.304844    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.304927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:13.304899    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 15:09:13.373846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:13.373806    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.374141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:13.374109    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:09:14.003061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:14.003014    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.003257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:14.003077    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:09:14.304910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:14.304825    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.304910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:14.304880    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 15:09:14.373807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:14.373764    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.374062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:14.374032    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:09:15.003492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:15.003450    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.003666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:15.003530    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:09:15.305068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:15.304984    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.305068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:15.305038    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 15:09:15.375384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:15.374802    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.375384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:15.374880    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:09:16.003135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:16.003097    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:09:16.003308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:16.003150    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:09:16.304635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:16.304550    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 15:09:16.304635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:16.304624    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 15:09:16.373945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:16.373904    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:09:16.374475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:16.373970    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:09:17.221040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.218571    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:17.221040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.220052    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:17.221968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.221912    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:09:17.310388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.310343    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:17.310806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.310776    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:17.311343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.311320    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:09:17.392393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.387189    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:17.392393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.388623    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:17.392393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:17.390180    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:09:18.314292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.314258    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"]
Nov 02 15:09:18.315399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.315358    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8761c5e7-df26-4f7b-9526-675d8c9995e3" containerName="revoke-keys"
Nov 02 15:09:18.315627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.315538    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8761c5e7-df26-4f7b-9526-675d8c9995e3" containerName="revoke-keys"
Nov 02 15:09:18.315900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.315883    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8761c5e7-df26-4f7b-9526-675d8c9995e3" containerName="revoke-keys"
Nov 02 15:09:18.323241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.323217    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.337062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.337018    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"]
Nov 02 15:09:18.372091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372060    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372161    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372224    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372589    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372639    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372789    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372816    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.372861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372845    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.373113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.372908    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5d4k\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.406955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.406931    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5113c2-openshift-default-dockercfg-kwr29\""
Nov 02 15:09:18.473771 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.473739    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.473915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.473798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.473981    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474049    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474143    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474187    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474226    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5d4k\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474226    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474312    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474311    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474426    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474605    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.474963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.474936    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.476325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.476306    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.476575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.476554    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.496278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.496251    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.496393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.496314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5d4k\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.651128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.651063    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:18.967898 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:18.967870    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af9fd91_65d3_486d_9d88_c098e6a85e84.slice/crio-bf8075f62ab3ac39ea7bd77739562245b393adc5fe5173a90154b2163934d4b4 WatchSource:0}: Error finding container bf8075f62ab3ac39ea7bd77739562245b393adc5fe5173a90154b2163934d4b4: Status 404 returned error can't find the container with id bf8075f62ab3ac39ea7bd77739562245b393adc5fe5173a90154b2163934d4b4
Nov 02 15:09:18.968079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.968058    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"]
Nov 02 15:09:18.970163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.970130    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:18.970246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.970192    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:18.970246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:18.970226    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:19.347223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:19.347132    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" event={"ID":"0af9fd91-65d3-486d-9d88-c098e6a85e84","Type":"ContainerStarted","Data":"23706eb38ed8df0a9c02d35f98b40a3609f53bffbedd768575286f961dc2eb33"}
Nov 02 15:09:19.347223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:19.347196    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" event={"ID":"0af9fd91-65d3-486d-9d88-c098e6a85e84","Type":"ContainerStarted","Data":"bf8075f62ab3ac39ea7bd77739562245b393adc5fe5173a90154b2163934d4b4"}
Nov 02 15:09:19.401319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:19.391261    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podStartSLOduration=1.391242048 podStartE2EDuration="1.391242048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:19.390560736 +0000 UTC m=+811.648630236" watchObservedRunningTime="2026-11-02 15:09:19.391242048 +0000 UTC m=+811.649311495"
Nov 02 15:09:19.651549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:19.651468    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:20.653177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:20.653140    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 15:09:20.653582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:20.653206    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 15:09:21.651639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:21.651603    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 15:09:21.651819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:21.651672    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 15:09:22.652443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:22.652407    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 15:09:22.652832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:22.652467    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 15:09:23.254261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.254221    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"]
Nov 02 15:09:23.263419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.258797    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.263419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.259686    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"]
Nov 02 15:09:23.273812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.273793    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"distinct-w2-739bf1-kserve-self-signed-certs\""
Nov 02 15:09:23.388381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.388335    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.388539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.388473    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.388608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.388555    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.388662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.388617    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.388750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.388731    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.388793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.388780    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5l6\" (UniqueName: \"kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.491406    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.491508    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.491573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5l6\" (UniqueName: \"kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.491756    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.491860    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.491913    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.492560    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.493664    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.493936    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.505327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.503732    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.511405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.508007    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.530386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.527259    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5l6\" (UniqueName: \"kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6\") pod \"distinct-w2-739bf1-kserve-74d6cf6f9-42rwg\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.576182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.576151    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:23.649121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.648703    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"]
Nov 02 15:09:23.651681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.651654    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 15:09:23.659432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.656291    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 15:09:23.660116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.660095    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.686474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.685400    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"]
Nov 02 15:09:23.795451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795341    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795444    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795509    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795560    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795591    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795673    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795711    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jq6b\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795758    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.795884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.795838    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897700    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897746    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897801    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897837    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897859    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897912    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.897976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897936    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jq6b\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.898168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.897974    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.898168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.898156    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.898398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.898375    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.898823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.898633    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.898823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.898762    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.899475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.899091    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.901053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.901032    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.901447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.901419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.940319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.940257    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jq6b\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.942947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.942464    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:23.943277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.943252    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"]
Nov 02 15:09:23.994436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:23.994416    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:24.364040 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:24.364007    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb17794_cb62_4a6e_bec6_45bc461b2a9b.slice/crio-1630e2fe572e00c9dbc23264f825d3d4ebb4dc155cab19c6abc39abf60f9cdeb WatchSource:0}: Error finding container 1630e2fe572e00c9dbc23264f825d3d4ebb4dc155cab19c6abc39abf60f9cdeb: Status 404 returned error can't find the container with id 1630e2fe572e00c9dbc23264f825d3d4ebb4dc155cab19c6abc39abf60f9cdeb
Nov 02 15:09:24.364270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.364249    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"]
Nov 02 15:09:24.367356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.366974    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:24.367356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.367035    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:24.367356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.367064    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:24.398146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.397639    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" event={"ID":"57bd7d06-b376-4dde-8a5e-f5d1149f83b2","Type":"ContainerStarted","Data":"17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303"}
Nov 02 15:09:24.398146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.397680    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" event={"ID":"57bd7d06-b376-4dde-8a5e-f5d1149f83b2","Type":"ContainerStarted","Data":"1ddaaa2bb00dd3aff5597bb9c41d068687c41323fdd4215c99b4cce52dc9b859"}
Nov 02 15:09:24.398994 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.398971    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" event={"ID":"9fb17794-cb62-4a6e-bec6-45bc461b2a9b","Type":"ContainerStarted","Data":"1630e2fe572e00c9dbc23264f825d3d4ebb4dc155cab19c6abc39abf60f9cdeb"}
Nov 02 15:09:24.443143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.442934    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podStartSLOduration=1.442914328 podStartE2EDuration="1.442914328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:24.437724024 +0000 UTC m=+816.695793474" watchObservedRunningTime="2026-11-02 15:09:24.442914328 +0000 UTC m=+816.700983779"
Nov 02 15:09:24.580528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.579511    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:24.652322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.652228    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 15:09:24.652322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:24.652309    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 15:09:25.408876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.408833    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" event={"ID":"9fb17794-cb62-4a6e-bec6-45bc461b2a9b","Type":"ContainerStarted","Data":"e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a"}
Nov 02 15:09:25.473404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.473309    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podStartSLOduration=2.473293595 podStartE2EDuration="2.473293595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:25.468676655 +0000 UTC m=+817.726746105" watchObservedRunningTime="2026-11-02 15:09:25.473293595 +0000 UTC m=+817.731363045"
Nov 02 15:09:25.688052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.688019    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:25.688448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.688402    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:25.689812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.689787    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:09:25.995185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.995104    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:25.996939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.996914    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:09:25.997073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:25.996966    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 15:09:26.995682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:26.995650    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:09:26.996062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:26.995701    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 15:09:27.995759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:27.995724    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:09:27.996118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:27.995784    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 15:09:28.998697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:28.998650    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:09:28.999196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:28.998726    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 15:09:29.995728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:29.995690    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:09:29.995885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:29.995751    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 15:09:30.995810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:30.995772    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:09:30.996313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:30.995828    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 15:09:32.139911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.139873    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"]
Nov 02 15:09:32.144088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.144067    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.176984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.176948    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"]
Nov 02 15:09:32.180025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.179999    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:09:32.182544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.181847    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:09:32.348725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.348683    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.349024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.348995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmpt\" (UniqueName: \"kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.349207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.349189    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.451325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.451279    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.451558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.451382    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmpt\" (UniqueName: \"kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.451558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.451450    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.452138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.452114    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.454431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.454404    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.475103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.475060    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmpt\" (UniqueName: \"kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt\") pod \"maas-api-revoke-sub-e35c1533-ccs6s\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:32.553136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.553101    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:32.553542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.553511    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:32.554271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.554245    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:09:32.762528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:32.762441    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:33.080432 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:33.080395    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70e113c_efe2_42fa_8473_ffa9667fe84e.slice/crio-cbed34c10a0634f171a45d5729cbc4bbc8b77b002562e7cc5f4ec04058e71319 WatchSource:0}: Error finding container cbed34c10a0634f171a45d5729cbc4bbc8b77b002562e7cc5f4ec04058e71319: Status 404 returned error can't find the container with id cbed34c10a0634f171a45d5729cbc4bbc8b77b002562e7cc5f4ec04058e71319
Nov 02 15:09:33.081494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:33.081464    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"]
Nov 02 15:09:33.521039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:33.521004    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s" event={"ID":"f70e113c-efe2-42fa-8473-ffa9667fe84e","Type":"ContainerStarted","Data":"b0f2ed061b51e72b4ea1efb4cbc6d66f0c5ad46fbd406024c38465cba25d3e2e"}
Nov 02 15:09:33.521509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:33.521057    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s" event={"ID":"f70e113c-efe2-42fa-8473-ffa9667fe84e","Type":"ContainerStarted","Data":"cbed34c10a0634f171a45d5729cbc4bbc8b77b002562e7cc5f4ec04058e71319"}
Nov 02 15:09:33.568329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:33.568274    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s" podStartSLOduration=1.568259837 podStartE2EDuration="1.568259837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:33.56368511 +0000 UTC m=+825.821754563" watchObservedRunningTime="2026-11-02 15:09:33.568259837 +0000 UTC m=+825.826329364"
Nov 02 15:09:33.577434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:33.577402    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:33.590880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:33.590856    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:34.530123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:34.530086    5041 generic.go:356] "Generic (PLEG): container finished" podID="f70e113c-efe2-42fa-8473-ffa9667fe84e" containerID="b0f2ed061b51e72b4ea1efb4cbc6d66f0c5ad46fbd406024c38465cba25d3e2e" exitCode=0
Nov 02 15:09:34.530589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:34.530181    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s" event={"ID":"f70e113c-efe2-42fa-8473-ffa9667fe84e","Type":"ContainerDied","Data":"b0f2ed061b51e72b4ea1efb4cbc6d66f0c5ad46fbd406024c38465cba25d3e2e"}
Nov 02 15:09:34.541829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:34.541803    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:09:35.741331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.741300    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:35.817607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.817521    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt\" (UniqueName: \"kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt\") pod \"f70e113c-efe2-42fa-8473-ffa9667fe84e\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") "
Nov 02 15:09:35.817607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.817575    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca\") pod \"f70e113c-efe2-42fa-8473-ffa9667fe84e\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") "
Nov 02 15:09:35.817607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.817607    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca\") pod \"f70e113c-efe2-42fa-8473-ffa9667fe84e\" (UID: \"f70e113c-efe2-42fa-8473-ffa9667fe84e\") "
Nov 02 15:09:35.818341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.818308    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca" pod "f70e113c-efe2-42fa-8473-ffa9667fe84e" (UID: "f70e113c-efe2-42fa-8473-ffa9667fe84e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:09:35.819896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.819866    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca" pod "f70e113c-efe2-42fa-8473-ffa9667fe84e" (UID: "f70e113c-efe2-42fa-8473-ffa9667fe84e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:09:35.820227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.820204    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt" pod "f70e113c-efe2-42fa-8473-ffa9667fe84e" (UID: "f70e113c-efe2-42fa-8473-ffa9667fe84e"). InnerVolumeSpecName "kube-api-access-nbmpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:09:35.919345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.919315    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbmpt\" (UniqueName: \"kubernetes.io/projected/f70e113c-efe2-42fa-8473-ffa9667fe84e-kube-api-access-nbmpt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:35.919345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.919342    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:35.919345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:35.919351    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70e113c-efe2-42fa-8473-ffa9667fe84e-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:09:36.546890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:36.546860    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"
Nov 02 15:09:36.546890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:36.546871    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s" event={"ID":"f70e113c-efe2-42fa-8473-ffa9667fe84e","Type":"ContainerDied","Data":"cbed34c10a0634f171a45d5729cbc4bbc8b77b002562e7cc5f4ec04058e71319"}
Nov 02 15:09:36.547101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:36.546924    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbed34c10a0634f171a45d5729cbc4bbc8b77b002562e7cc5f4ec04058e71319"
Nov 02 15:09:36.768532 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:09:36.768497    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:09:41.070786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.070749    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"]
Nov 02 15:09:41.072206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.072175    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f70e113c-efe2-42fa-8473-ffa9667fe84e" containerName="revoke-keys"
Nov 02 15:09:41.072206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.072205    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70e113c-efe2-42fa-8473-ffa9667fe84e" containerName="revoke-keys"
Nov 02 15:09:41.072442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.072393    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70e113c-efe2-42fa-8473-ffa9667fe84e" containerName="revoke-keys"
Nov 02 15:09:41.076428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.076404    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.102828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.102792    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"openshift-service-ca.crt\""
Nov 02 15:09:41.103065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.102989    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"default-dockercfg-2cv7c\""
Nov 02 15:09:41.103211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.102974    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"distinct-2-w4-7ecfe1-kserve-self-signed-certs\""
Nov 02 15:09:41.103211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.102814    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"kube-root-ca.crt\""
Nov 02 15:09:41.110138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.110108    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"]
Nov 02 15:09:41.196511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.196475    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5wh7\" (UniqueName: \"kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.196718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.196553    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.196718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.196673    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.196848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.196718    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.196848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.196824    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.196954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.196854    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.298717    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.298809    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.298847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.298921    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.298969    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.299102    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5wh7\" (UniqueName: \"kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.299904    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.300174    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.300850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.300469    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.336825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.336733    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.338835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.338718    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5wh7\" (UniqueName: \"kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.346510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.342507    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs\") pod \"distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.390679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.390644    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:41.703938 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:41.703909    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49c1029f_d9e4_4932_922f_8336beee4182.slice/crio-c4539b8792491b3a81415ca63b7fbf29d4c47cddecd36b036710ec29e661133d WatchSource:0}: Error finding container c4539b8792491b3a81415ca63b7fbf29d4c47cddecd36b036710ec29e661133d: Status 404 returned error can't find the container with id c4539b8792491b3a81415ca63b7fbf29d4c47cddecd36b036710ec29e661133d
Nov 02 15:09:41.704133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:41.704109    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"]
Nov 02 15:09:42.604023 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:42.603987    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" event={"ID":"49c1029f-d9e4-4932-922f-8336beee4182","Type":"ContainerStarted","Data":"1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0"}
Nov 02 15:09:42.604513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:42.604033    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" event={"ID":"49c1029f-d9e4-4932-922f-8336beee4182","Type":"ContainerStarted","Data":"c4539b8792491b3a81415ca63b7fbf29d4c47cddecd36b036710ec29e661133d"}
Nov 02 15:09:42.648312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:42.648256    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podStartSLOduration=1.6482429619999999 podStartE2EDuration="1.648242962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:42.644081243 +0000 UTC m=+834.902150692" watchObservedRunningTime="2026-11-02 15:09:42.648242962 +0000 UTC m=+834.906312416"
Nov 02 15:09:43.391692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:43.391660    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:51.138171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.138133    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"]
Nov 02 15:09:51.141867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.141844    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.156191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.156165    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"kube-root-ca.crt\""
Nov 02 15:09:51.167674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.167649    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"openshift-service-ca.crt\""
Nov 02 15:09:51.167784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.167693    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"distinct-2-w5-1cc1b2-kserve-self-signed-certs\""
Nov 02 15:09:51.167851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.167786    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"default-dockercfg-l4n54\""
Nov 02 15:09:51.173009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.172987    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"]
Nov 02 15:09:51.224323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.224288    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.224512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.224332    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.224512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.224451    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.224512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.224472    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8tn\" (UniqueName: \"kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.224683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.224593    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.224683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.224649    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.330918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.330993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.331196    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.331231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8tn\" (UniqueName: \"kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.331348    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.331479    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.333463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.332039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.344399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.343857    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.344399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.344146    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.370314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.370278    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.387762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.387523    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.389328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.389268    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8tn\" (UniqueName: \"kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn\") pod \"distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.394122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.394100    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:51.430552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.430515    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:51.458219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.458190    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:51.692815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.692782    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:09:51.776255 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:51.776223    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f5d7402_2fc1_44b7_b082_19b2ecf56b26.slice/crio-8476eb2358bb623fcbe3ba9997025c9ecd177c18e9fc4cbb820dd9ae47034bc9 WatchSource:0}: Error finding container 8476eb2358bb623fcbe3ba9997025c9ecd177c18e9fc4cbb820dd9ae47034bc9: Status 404 returned error can't find the container with id 8476eb2358bb623fcbe3ba9997025c9ecd177c18e9fc4cbb820dd9ae47034bc9
Nov 02 15:09:51.776721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:51.776696    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"]
Nov 02 15:09:52.691131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:52.691085    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" event={"ID":"3f5d7402-2fc1-44b7-b082-19b2ecf56b26","Type":"ContainerStarted","Data":"d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60"}
Nov 02 15:09:52.691623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:52.691138    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" event={"ID":"3f5d7402-2fc1-44b7-b082-19b2ecf56b26","Type":"ContainerStarted","Data":"8476eb2358bb623fcbe3ba9997025c9ecd177c18e9fc4cbb820dd9ae47034bc9"}
Nov 02 15:09:52.737286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:52.737234    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podStartSLOduration=1.7372210319999999 podStartE2EDuration="1.737221032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:52.733725871 +0000 UTC m=+844.991795320" watchObservedRunningTime="2026-11-02 15:09:52.737221032 +0000 UTC m=+844.995290482"
Nov 02 15:09:53.473387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:53.467506    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:09:54.064589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.064551    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"]
Nov 02 15:09:54.102395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.094338    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.148504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.148471    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"]
Nov 02 15:09:54.185960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.185914    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"]
Nov 02 15:09:54.194605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.191894    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.197085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197031    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.197344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197322    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.197508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197490    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.197645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197630    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q4x\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.197743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197729    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.197859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197845    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.197968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.197955    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.198121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.198108    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.198301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.198266    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.344299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.344147    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.345349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.345323    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.346039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346015    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.346221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346202    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.346357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346341    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.346556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346537    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.346696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.346813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346797    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.346935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.346919    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.347055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.347039    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.347237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.347219    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.347450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.347427    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.347621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.347602    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vg66\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.347770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.347753    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.347911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.347895    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.348075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.348058    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q4x\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.348200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.348184    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.348344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.348327    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.350923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.345941    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.352404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.351774    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.352545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.351809    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.352650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.352322    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.352752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.352416    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.359695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.357905    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"]
Nov 02 15:09:54.360062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.360043    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.386960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.386907    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.392609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.392561    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q4x\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.394877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.394858    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.449474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.449440    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.449638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.449608    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450174    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450252    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450296    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450326    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450657    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.450810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450756    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.451584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450966    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.452904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.450357    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.452904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.451767    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.452904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.451823    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vg66\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.452904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.452861    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.453465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.453415    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.453721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.453682    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.473880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.473853    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.475195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.475166    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vg66\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.486886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.486855    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:54.535657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.535575    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:54.846411 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:09:54.843481    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3d7955_073c_4271_83f7_d18f2962b0de.slice/crio-ea421e9866eb0c2b297e95a4b7452832c4575e7eef98b785f2357fe076c691d9 WatchSource:0}: Error finding container ea421e9866eb0c2b297e95a4b7452832c4575e7eef98b785f2357fe076c691d9: Status 404 returned error can't find the container with id ea421e9866eb0c2b297e95a4b7452832c4575e7eef98b785f2357fe076c691d9
Nov 02 15:09:54.858926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.854764    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.858926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.855108    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.858926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.855160    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.875600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.875568    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"]
Nov 02 15:09:54.931415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.931279    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"]
Nov 02 15:09:54.932494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.931963    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.932494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.932045    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.932494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:54.932088    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:55.765157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:55.765116    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" event={"ID":"0e259152-910d-43c6-a8fb-5b231bb2632c","Type":"ContainerStarted","Data":"492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7"}
Nov 02 15:09:55.776672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:55.776629    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" event={"ID":"0e259152-910d-43c6-a8fb-5b231bb2632c","Type":"ContainerStarted","Data":"af3c2fa614c6750b79d52cfc07bd5d85534d7bd0ae994bcd2e7ddc21356aaf19"}
Nov 02 15:09:55.779662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:55.779622    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" event={"ID":"0e3d7955-073c-4271-83f7-d18f2962b0de","Type":"ContainerStarted","Data":"c18b802140dbdcf147a0d57c3824b1f6d61cae3b2030a5345327ee89b2bf7309"}
Nov 02 15:09:55.779837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:55.779822    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" event={"ID":"0e3d7955-073c-4271-83f7-d18f2962b0de","Type":"ContainerStarted","Data":"ea421e9866eb0c2b297e95a4b7452832c4575e7eef98b785f2357fe076c691d9"}
Nov 02 15:09:55.873699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:55.873635    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podStartSLOduration=1.873618837 podStartE2EDuration="1.873618837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:55.845619864 +0000 UTC m=+848.103689315" watchObservedRunningTime="2026-11-02 15:09:55.873618837 +0000 UTC m=+848.131688287"
Nov 02 15:09:55.894245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:55.894191    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podStartSLOduration=1.894176994 podStartE2EDuration="1.894176994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:55.892546507 +0000 UTC m=+848.150615959" watchObservedRunningTime="2026-11-02 15:09:55.894176994 +0000 UTC m=+848.152246510"
Nov 02 15:09:56.487972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:56.487934    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:09:56.489987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:56.489958    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:09:56.490124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:56.490008    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 15:09:56.536395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:56.535787    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:09:56.545386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:56.544063    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:09:56.545386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:56.544129    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:09:57.488026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:57.487992    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:09:57.488445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:57.488045    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 15:09:57.548387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:57.547172    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:09:57.548387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:57.547227    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:09:58.487212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:58.487175    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:09:58.487392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:58.487222    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 15:09:58.537397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:58.536827    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:09:58.537397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:58.536897    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:09:59.487901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:59.487864    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:09:59.488112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:59.487923    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 15:09:59.541387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:59.540636    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:09:59.541387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:09:59.540690    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:10:00.487893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:00.487853    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:10:00.488101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:00.487912    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 15:10:00.545384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:00.544705    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:10:00.545384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:00.544768    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:10:01.458542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.458497    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:10:01.477493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.477466    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:10:01.487923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.487888    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:10:01.488068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.487958    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 15:10:01.544773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.544735    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:10:01.545065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.545030    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:10:01.905752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:01.905658    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:10:02.491504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:02.491475    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:10:02.491967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:02.491942    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:10:02.492639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:02.492617    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:10:02.540079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:02.540051    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:10:02.540346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:02.540289    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:10:02.542318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:02.541598    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:10:06.769218 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:10:06.769182    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:10:09.070765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.070731    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"]
Nov 02 15:10:09.074925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.074909    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.103065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.103005    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"]
Nov 02 15:10:09.232750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.232717    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.232750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.232750    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.232794    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdwq\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.232997    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.233083    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.233131    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.233225    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.233253    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.233300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.233281    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335093    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335166    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335219    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335242    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335267    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335417    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335464    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335541    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdwq\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335556    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335649    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.335804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335722    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.336027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335847    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.336027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.335900    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.336170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.336151    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.337701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.337675    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.337833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.337812    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.357133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.357109    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.357252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.357186    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdwq\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.390215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.390190    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:09.715066 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:09.715017    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ebbc3_45aa_4a3e_a1ed_19fc20328400.slice/crio-c96795415b87280a9cd0b3924ffacd4f9ececee215f2d107dab5f65a25b0afe0 WatchSource:0}: Error finding container c96795415b87280a9cd0b3924ffacd4f9ececee215f2d107dab5f65a25b0afe0: Status 404 returned error can't find the container with id c96795415b87280a9cd0b3924ffacd4f9ececee215f2d107dab5f65a25b0afe0
Nov 02 15:10:09.715767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.715702    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"]
Nov 02 15:10:09.717767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.717735    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:09.717845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.717800    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:09.717845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:09.717833    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:10.007713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:10.007615    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" event={"ID":"481ebbc3-45aa-4a3e-a1ed-19fc20328400","Type":"ContainerStarted","Data":"98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a"}
Nov 02 15:10:10.007713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:10.007681    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" event={"ID":"481ebbc3-45aa-4a3e-a1ed-19fc20328400","Type":"ContainerStarted","Data":"c96795415b87280a9cd0b3924ffacd4f9ececee215f2d107dab5f65a25b0afe0"}
Nov 02 15:10:10.053689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:10.053640    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podStartSLOduration=1.053626868 podStartE2EDuration="1.053626868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:10.048598987 +0000 UTC m=+862.306668438" watchObservedRunningTime="2026-11-02 15:10:10.053626868 +0000 UTC m=+862.311696318"
Nov 02 15:10:10.391234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:10.391147    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:10.392925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:10.392902    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:10:10.392995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:10.392949    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 15:10:11.391345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:11.391309    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:10:11.391851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:11.391394    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 15:10:12.391148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:12.391110    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:10:12.391333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:12.391163    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 15:10:13.390584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:13.390547    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:10:13.390952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:13.390605    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 15:10:14.390646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:14.390611    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:10:14.391018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:14.390674    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 15:10:15.391007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:15.390967    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:10:15.391502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:15.391024    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 15:10:16.672131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:16.672093    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:16.673112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:16.672748    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:16.673976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:16.673945    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:10:25.932845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.932809    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"]
Nov 02 15:10:25.939275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.939254    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:25.952577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.952552    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfnvh\" (UniqueName: \"kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:25.952670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.952612    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:25.952804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.952780    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:25.958549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.958525    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"]
Nov 02 15:10:25.961524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.961498    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:10:25.961524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:25.961516    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:10:26.053857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.053830    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfnvh\" (UniqueName: \"kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.053980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.053887    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.053980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.053947    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.054538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.054520    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.056402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.056380    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.074776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.074751    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfnvh\" (UniqueName: \"kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh\") pod \"maas-api-revoke-sub-4f8d8eb2-v792h\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.253925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.253852    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:26.567420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:26.567390    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"]
Nov 02 15:10:26.567573 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:26.567391    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936b72c8_4d36_430c_a469_01d2d2f85120.slice/crio-50a2cd30958b6f131f398a15dccd099b6db301625b136cd90cfad24eb9f95e55 WatchSource:0}: Error finding container 50a2cd30958b6f131f398a15dccd099b6db301625b136cd90cfad24eb9f95e55: Status 404 returned error can't find the container with id 50a2cd30958b6f131f398a15dccd099b6db301625b136cd90cfad24eb9f95e55
Nov 02 15:10:27.227504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.227473    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"]
Nov 02 15:10:27.232611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.232584    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.236163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.235660    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h" event={"ID":"936b72c8-4d36-430c-a469-01d2d2f85120","Type":"ContainerStarted","Data":"730b0d20e7d4df35082e175322dd39f4e0c13e1983f2c2f72fe2b13d8e46f3e3"}
Nov 02 15:10:27.236163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.235716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h" event={"ID":"936b72c8-4d36-430c-a469-01d2d2f85120","Type":"ContainerStarted","Data":"50a2cd30958b6f131f398a15dccd099b6db301625b136cd90cfad24eb9f95e55"}
Nov 02 15:10:27.256855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.256830    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"kube-root-ca.crt\""
Nov 02 15:10:27.256855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.256839    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"default-dockercfg-ccmk6\""
Nov 02 15:10:27.256997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.256839    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"openshift-service-ca.crt\""
Nov 02 15:10:27.256997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.256906    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"distinct-2-w1-5dd8e9-kserve-self-signed-certs\""
Nov 02 15:10:27.261852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.261829    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"]
Nov 02 15:10:27.268176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.268153    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.268331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.268195    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.268404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.268320    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.268503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.268484    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.268540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.268526    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.268620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.268604    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p747q\" (UniqueName: \"kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.310431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.310357    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h" podStartSLOduration=2.310342039 podStartE2EDuration="2.310342039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:27.307867332 +0000 UTC m=+879.565936781" watchObservedRunningTime="2026-11-02 15:10:27.310342039 +0000 UTC m=+879.568411489"
Nov 02 15:10:27.369714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.369683    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p747q\" (UniqueName: \"kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.369820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.369801    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.369870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.369839    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.369870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.369865    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.370104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.370072    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.370230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.370127    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.370296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.370272    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.370511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.370488    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.370584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.370509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.372230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.372211    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.372539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.372521    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.390140 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.390110    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p747q\" (UniqueName: \"kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q\") pod \"distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.550263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.550169    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:27.888052 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:27.888021    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3c359bc_c443_43b2_b9a7_2f12d2c42b8b.slice/crio-c78aa77ac37d1b8b3e1562daca2d7d869f7ce80e6d02218fa3b5967360fe4ae7 WatchSource:0}: Error finding container c78aa77ac37d1b8b3e1562daca2d7d869f7ce80e6d02218fa3b5967360fe4ae7: Status 404 returned error can't find the container with id c78aa77ac37d1b8b3e1562daca2d7d869f7ce80e6d02218fa3b5967360fe4ae7
Nov 02 15:10:27.890349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:27.890324    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"]
Nov 02 15:10:28.246742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:28.246706    5041 generic.go:356] "Generic (PLEG): container finished" podID="936b72c8-4d36-430c-a469-01d2d2f85120" containerID="730b0d20e7d4df35082e175322dd39f4e0c13e1983f2c2f72fe2b13d8e46f3e3" exitCode=0
Nov 02 15:10:28.247172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:28.246793    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h" event={"ID":"936b72c8-4d36-430c-a469-01d2d2f85120","Type":"ContainerDied","Data":"730b0d20e7d4df35082e175322dd39f4e0c13e1983f2c2f72fe2b13d8e46f3e3"}
Nov 02 15:10:28.248650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:28.248626    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" event={"ID":"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b","Type":"ContainerStarted","Data":"a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc"}
Nov 02 15:10:28.248783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:28.248655    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" event={"ID":"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b","Type":"ContainerStarted","Data":"c78aa77ac37d1b8b3e1562daca2d7d869f7ce80e6d02218fa3b5967360fe4ae7"}
Nov 02 15:10:28.322108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:28.322062    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podStartSLOduration=1.322048642 podStartE2EDuration="1.322048642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:28.318534337 +0000 UTC m=+880.576603789" watchObservedRunningTime="2026-11-02 15:10:28.322048642 +0000 UTC m=+880.580118091"
Nov 02 15:10:28.551864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:28.551789    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:29.477728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.477704    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:29.523489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.523423    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh\" (UniqueName: \"kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh\") pod \"936b72c8-4d36-430c-a469-01d2d2f85120\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") "
Nov 02 15:10:29.523660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.523518    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca\") pod \"936b72c8-4d36-430c-a469-01d2d2f85120\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") "
Nov 02 15:10:29.523745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.523727    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca\") pod \"936b72c8-4d36-430c-a469-01d2d2f85120\" (UID: \"936b72c8-4d36-430c-a469-01d2d2f85120\") "
Nov 02 15:10:29.524172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.524142    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca" pod "936b72c8-4d36-430c-a469-01d2d2f85120" (UID: "936b72c8-4d36-430c-a469-01d2d2f85120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:29.524344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.524293    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:29.526830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.526805    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh" pod "936b72c8-4d36-430c-a469-01d2d2f85120" (UID: "936b72c8-4d36-430c-a469-01d2d2f85120"). InnerVolumeSpecName "kube-api-access-lfnvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:29.527758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.527736    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca" pod "936b72c8-4d36-430c-a469-01d2d2f85120" (UID: "936b72c8-4d36-430c-a469-01d2d2f85120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:29.625734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.625713    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936b72c8-4d36-430c-a469-01d2d2f85120-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:29.625734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:29.625735    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfnvh\" (UniqueName: \"kubernetes.io/projected/936b72c8-4d36-430c-a469-01d2d2f85120-kube-api-access-lfnvh\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:30.267981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:30.267944    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h" event={"ID":"936b72c8-4d36-430c-a469-01d2d2f85120","Type":"ContainerDied","Data":"50a2cd30958b6f131f398a15dccd099b6db301625b136cd90cfad24eb9f95e55"}
Nov 02 15:10:30.268165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:30.268001    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a2cd30958b6f131f398a15dccd099b6db301625b136cd90cfad24eb9f95e55"
Nov 02 15:10:30.268165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:30.267957    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"
Nov 02 15:10:36.769204 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:10:36.769157    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:10:37.551521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.551488    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:37.563994 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.563969    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:37.865165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.865095    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"]
Nov 02 15:10:37.865780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.865765    5041 cpu_manager.go:414] "containerMap: removing container" podUID="936b72c8-4d36-430c-a469-01d2d2f85120" containerName="revoke-keys"
Nov 02 15:10:37.865825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.865781    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936b72c8-4d36-430c-a469-01d2d2f85120" containerName="revoke-keys"
Nov 02 15:10:37.865891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.865882    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="936b72c8-4d36-430c-a469-01d2d2f85120" containerName="revoke-keys"
Nov 02 15:10:37.869575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.869558    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:37.881845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.881820    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:10:37.882002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.881990    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:10:37.893063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.893039    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"]
Nov 02 15:10:37.971637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.971599    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9bd\" (UniqueName: \"kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:37.971854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.971829    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:37.972517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:37.971940    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.072916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.072866    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9bd\" (UniqueName: \"kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.073046 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.073031    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.073133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.073116    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.073895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.073864    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.075544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.075516    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.093766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.093741    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9bd\" (UniqueName: \"kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd\") pod \"maas-api-revoke-sub-6742598b-574j4\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.184264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.184242    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:38.365256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.365230    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:38.492936 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:38.492905    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc163f924_dc2f_4416_8d58_a5e0319dff6e.slice/crio-1081f1d7253f68ea42888156a450656b13b0d660c93c197a7e6d23b934356fe1 WatchSource:0}: Error finding container 1081f1d7253f68ea42888156a450656b13b0d660c93c197a7e6d23b934356fe1: Status 404 returned error can't find the container with id 1081f1d7253f68ea42888156a450656b13b0d660c93c197a7e6d23b934356fe1
Nov 02 15:10:38.493060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.492986    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"]
Nov 02 15:10:38.818381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.818342    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"]
Nov 02 15:10:38.823936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.823920    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:38.844632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.844605    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"]
Nov 02 15:10:38.875038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.875008    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"]
Nov 02 15:10:38.880222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.879912    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.915489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.915456    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"]
Nov 02 15:10:38.986390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.986337    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:38.986541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.986500    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.986541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.986534    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82p2\" (UniqueName: \"kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:38.986691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.986669    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.986753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.986712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.987052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987027    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7h6\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.987141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987106    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.987228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987211    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.987289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987243    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:38.987439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987418    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.987532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987457    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:38.987597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:38.987543    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.088987    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089028    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089068    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089116    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089138    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82p2\" (UniqueName: \"kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089187    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089205    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089265    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7h6\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089317    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089353    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089386    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.089732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089441    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089443    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089682    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089778    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089436    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.089939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.089807    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.090493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.090469    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.091860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.091796    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.092018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.092008    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.092700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.092673    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.110753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.110730    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.110841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.110809    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82p2\" (UniqueName: \"kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2\") pod \"maas-api-revoke-sub-9d90b7fe-n6mh5\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.111156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.111140    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7h6\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.140045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.140024    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:39.201481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.201408    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:39.365681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.365574    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4" event={"ID":"c163f924-dc2f-4416-8d58-a5e0319dff6e","Type":"ContainerStarted","Data":"67c70e56c9761f76b036a203caca5dd4f107168e022334ee54a48fe085ee21fc"}
Nov 02 15:10:39.365681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.365633    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4" event={"ID":"c163f924-dc2f-4416-8d58-a5e0319dff6e","Type":"ContainerStarted","Data":"1081f1d7253f68ea42888156a450656b13b0d660c93c197a7e6d23b934356fe1"}
Nov 02 15:10:39.424924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.424016    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4" podStartSLOduration=2.423998809 podStartE2EDuration="2.423998809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:39.410922041 +0000 UTC m=+891.668991494" watchObservedRunningTime="2026-11-02 15:10:39.423998809 +0000 UTC m=+891.682068261"
Nov 02 15:10:39.447043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.447006    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"]
Nov 02 15:10:39.464641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.456640    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.478684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.478655    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"]
Nov 02 15:10:39.486445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.486334    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"]
Nov 02 15:10:39.510318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.510290    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.510482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.510360    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.510562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.510535    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblqm\" (UniqueName: \"kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.553009 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:39.552980    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3604113c_a16f_42b0_b65e_f0e3456c6156.slice/crio-79b982de66ba95acbe232ff92d5b835594453ce5e264d0802e7c120f96fcdf8e WatchSource:0}: Error finding container 79b982de66ba95acbe232ff92d5b835594453ce5e264d0802e7c120f96fcdf8e: Status 404 returned error can't find the container with id 79b982de66ba95acbe232ff92d5b835594453ce5e264d0802e7c120f96fcdf8e
Nov 02 15:10:39.553716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.553692    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"]
Nov 02 15:10:39.555324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.555297    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.555423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.555355    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.555468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.555430    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.611247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.611222    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblqm\" (UniqueName: \"kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.611393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.611355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.611469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.611451    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.612071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.612049    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.613803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.613786    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.642993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.642969    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblqm\" (UniqueName: \"kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm\") pod \"maas-api-revoke-sub-f401c0bd-n48p8\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:39.790793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:39.790765    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:40.112082 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:40.112049    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8853fb03_fa8a_4723_b811_0e9de3866d37.slice/crio-99985a0c1bc6ff6538f610232b84f1cff594d3f8b2cfadda1e575e47a1869b41 WatchSource:0}: Error finding container 99985a0c1bc6ff6538f610232b84f1cff594d3f8b2cfadda1e575e47a1869b41: Status 404 returned error can't find the container with id 99985a0c1bc6ff6538f610232b84f1cff594d3f8b2cfadda1e575e47a1869b41
Nov 02 15:10:40.112398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.112242    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"]
Nov 02 15:10:40.384844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.384730    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5" event={"ID":"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110","Type":"ContainerStarted","Data":"6aeff1068468130f639d125af6f9073f32c110d4b3ff17fd01cc920478a307e9"}
Nov 02 15:10:40.385021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.384842    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5" event={"ID":"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110","Type":"ContainerStarted","Data":"b818830d84654c28bb16b0a959e49ae20f3a4d0a6df6badd52ace58cbd87bd6c"}
Nov 02 15:10:40.386545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.386517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" event={"ID":"3604113c-a16f-42b0-b65e-f0e3456c6156","Type":"ContainerStarted","Data":"8d4c3aa74446848dfeead771c8ef4dc2a7e64375383f27470a9f3cf9c653f828"}
Nov 02 15:10:40.386671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.386565    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" event={"ID":"3604113c-a16f-42b0-b65e-f0e3456c6156","Type":"ContainerStarted","Data":"79b982de66ba95acbe232ff92d5b835594453ce5e264d0802e7c120f96fcdf8e"}
Nov 02 15:10:40.388128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.388102    5041 generic.go:356] "Generic (PLEG): container finished" podID="c163f924-dc2f-4416-8d58-a5e0319dff6e" containerID="67c70e56c9761f76b036a203caca5dd4f107168e022334ee54a48fe085ee21fc" exitCode=0
Nov 02 15:10:40.388215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.388172    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4" event={"ID":"c163f924-dc2f-4416-8d58-a5e0319dff6e","Type":"ContainerDied","Data":"67c70e56c9761f76b036a203caca5dd4f107168e022334ee54a48fe085ee21fc"}
Nov 02 15:10:40.389444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.389424    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8" event={"ID":"8853fb03-fa8a-4723-b811-0e9de3866d37","Type":"ContainerStarted","Data":"99985a0c1bc6ff6538f610232b84f1cff594d3f8b2cfadda1e575e47a1869b41"}
Nov 02 15:10:40.444822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.444766    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5" podStartSLOduration=2.444749303 podStartE2EDuration="2.444749303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.420771526 +0000 UTC m=+892.678840977" watchObservedRunningTime="2026-11-02 15:10:40.444749303 +0000 UTC m=+892.702818770"
Nov 02 15:10:40.504381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:40.504320    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podStartSLOduration=2.504306199 podStartE2EDuration="2.504306199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.498734454 +0000 UTC m=+892.756803905" watchObservedRunningTime="2026-11-02 15:10:40.504306199 +0000 UTC m=+892.762375649"
Nov 02 15:10:41.207342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.204720    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:41.210589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.210052    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:41.210589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.210116    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:41.400242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.400208    5041 generic.go:356] "Generic (PLEG): container finished" podID="8853fb03-fa8a-4723-b811-0e9de3866d37" containerID="195fdd62627f1f9d8b9ffc6c2979702725a83f7e41086620f2c3a33705ebbed5" exitCode=0
Nov 02 15:10:41.400443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.400289    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8" event={"ID":"8853fb03-fa8a-4723-b811-0e9de3866d37","Type":"ContainerDied","Data":"195fdd62627f1f9d8b9ffc6c2979702725a83f7e41086620f2c3a33705ebbed5"}
Nov 02 15:10:41.402019 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.401989    5041 generic.go:356] "Generic (PLEG): container finished" podID="f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" containerID="6aeff1068468130f639d125af6f9073f32c110d4b3ff17fd01cc920478a307e9" exitCode=0
Nov 02 15:10:41.402135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.402072    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5" event={"ID":"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110","Type":"ContainerDied","Data":"6aeff1068468130f639d125af6f9073f32c110d4b3ff17fd01cc920478a307e9"}
Nov 02 15:10:41.642725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.642694    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:41.657307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.657279    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca\") pod \"c163f924-dc2f-4416-8d58-a5e0319dff6e\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") "
Nov 02 15:10:41.657532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.657309    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca\") pod \"c163f924-dc2f-4416-8d58-a5e0319dff6e\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") "
Nov 02 15:10:41.657532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.657335    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd\" (UniqueName: \"kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd\") pod \"c163f924-dc2f-4416-8d58-a5e0319dff6e\" (UID: \"c163f924-dc2f-4416-8d58-a5e0319dff6e\") "
Nov 02 15:10:41.657660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.657624    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca" pod "c163f924-dc2f-4416-8d58-a5e0319dff6e" (UID: "c163f924-dc2f-4416-8d58-a5e0319dff6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:41.657783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.657768    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:41.659473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.659441    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd" pod "c163f924-dc2f-4416-8d58-a5e0319dff6e" (UID: "c163f924-dc2f-4416-8d58-a5e0319dff6e"). InnerVolumeSpecName "kube-api-access-jp9bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:41.659572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.659468    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca" pod "c163f924-dc2f-4416-8d58-a5e0319dff6e" (UID: "c163f924-dc2f-4416-8d58-a5e0319dff6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:41.759026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.758953    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c163f924-dc2f-4416-8d58-a5e0319dff6e-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:41.759026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:41.758983    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9bd\" (UniqueName: \"kubernetes.io/projected/c163f924-dc2f-4416-8d58-a5e0319dff6e-kube-api-access-jp9bd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:42.202426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.202387    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:42.202589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.202450    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:42.410999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.410976    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"
Nov 02 15:10:42.411463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.410970    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4" event={"ID":"c163f924-dc2f-4416-8d58-a5e0319dff6e","Type":"ContainerDied","Data":"1081f1d7253f68ea42888156a450656b13b0d660c93c197a7e6d23b934356fe1"}
Nov 02 15:10:42.411463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.411080    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1081f1d7253f68ea42888156a450656b13b0d660c93c197a7e6d23b934356fe1"
Nov 02 15:10:42.685160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.685136    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:42.687186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.687163    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:42.772798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.772709    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2\" (UniqueName: \"kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2\") pod \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") "
Nov 02 15:10:42.772798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.772744    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm\" (UniqueName: \"kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm\") pod \"8853fb03-fa8a-4723-b811-0e9de3866d37\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") "
Nov 02 15:10:42.773141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.772834    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca\") pod \"8853fb03-fa8a-4723-b811-0e9de3866d37\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") "
Nov 02 15:10:42.773141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.772891    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca\") pod \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") "
Nov 02 15:10:42.773141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.772914    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca\") pod \"8853fb03-fa8a-4723-b811-0e9de3866d37\" (UID: \"8853fb03-fa8a-4723-b811-0e9de3866d37\") "
Nov 02 15:10:42.773141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.772961    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca\") pod \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\" (UID: \"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110\") "
Nov 02 15:10:42.773856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.773720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca" pod "8853fb03-fa8a-4723-b811-0e9de3866d37" (UID: "8853fb03-fa8a-4723-b811-0e9de3866d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:42.774304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.774283    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca" pod "f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" (UID: "f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:42.782823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.782791    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2" pod "f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" (UID: "f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110"). InnerVolumeSpecName "kube-api-access-b82p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:42.783286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.783258    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca" pod "f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" (UID: "f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:42.786735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.786708    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm" pod "8853fb03-fa8a-4723-b811-0e9de3866d37" (UID: "8853fb03-fa8a-4723-b811-0e9de3866d37"). InnerVolumeSpecName "kube-api-access-hblqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:42.786813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.786784    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca" pod "8853fb03-fa8a-4723-b811-0e9de3866d37" (UID: "8853fb03-fa8a-4723-b811-0e9de3866d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:42.874825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.874790    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82p2\" (UniqueName: \"kubernetes.io/projected/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-kube-api-access-b82p2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:42.874825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.874819    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblqm\" (UniqueName: \"kubernetes.io/projected/8853fb03-fa8a-4723-b811-0e9de3866d37-kube-api-access-hblqm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:42.874825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.874830    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:42.875119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.874839    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:42.875119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.874850    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8853fb03-fa8a-4723-b811-0e9de3866d37-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:42.875119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:42.874859    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:43.202517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.202482    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:43.202677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.202533    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:43.441490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.441448    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"
Nov 02 15:10:43.441889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.441491    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8" event={"ID":"8853fb03-fa8a-4723-b811-0e9de3866d37","Type":"ContainerDied","Data":"99985a0c1bc6ff6538f610232b84f1cff594d3f8b2cfadda1e575e47a1869b41"}
Nov 02 15:10:43.441889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.441553    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99985a0c1bc6ff6538f610232b84f1cff594d3f8b2cfadda1e575e47a1869b41"
Nov 02 15:10:43.443205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.443184    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"
Nov 02 15:10:43.443205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.443199    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5" event={"ID":"f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110","Type":"ContainerDied","Data":"b818830d84654c28bb16b0a959e49ae20f3a4d0a6df6badd52ace58cbd87bd6c"}
Nov 02 15:10:43.443462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:43.443236    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b818830d84654c28bb16b0a959e49ae20f3a4d0a6df6badd52ace58cbd87bd6c"
Nov 02 15:10:44.201732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:44.201696    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:44.201920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:44.201754    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:45.214387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:45.205812    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:45.214387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:45.205879    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:45.821388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:45.809249    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.049388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.046489    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.210656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.210615    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:46.210978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.210925    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:46.486626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.465010    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.585389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.584642    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.721358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.721314    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.814144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.814048    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.824589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.824560    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:10:46.906149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.906105    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.936388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:46.930854    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.094608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.094513    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.095382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.095340    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:10:47.306991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.306952    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:47.328387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.328328    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:47.526383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.512497    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.576244 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.552467    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:47.603393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.590709    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.603393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.591397    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:10:47.757385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.727460    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:47.757385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.727566    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.757385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.727864    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.757385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.728215    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:10:47.757385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.745871    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:10:47.757385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.753438    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.825440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.823776    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.913581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.913484    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.948383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:47.947642    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:10:48.069388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.069086    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.258191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.258153    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:10:48.258464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.258435    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:10:48.311348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.311292    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:10:48.322132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.322096    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:48.322406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.322342    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.326070 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.326041    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:48.326184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.326090    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.333205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.331541    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:10:48.333205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.331606    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:10:48.333205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.331658    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:10:48.333205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.331699    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:10:48.333205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.331734    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:10:48.333205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.331769    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:10:48.341496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.341242    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.352125    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"]
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353078    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8853fb03-fa8a-4723-b811-0e9de3866d37" containerName="revoke-keys"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353096    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8853fb03-fa8a-4723-b811-0e9de3866d37" containerName="revoke-keys"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353132    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c163f924-dc2f-4416-8d58-a5e0319dff6e" containerName="revoke-keys"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353139    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c163f924-dc2f-4416-8d58-a5e0319dff6e" containerName="revoke-keys"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353157    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" containerName="revoke-keys"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353165    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" containerName="revoke-keys"
Nov 02 15:10:48.353379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353351    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8853fb03-fa8a-4723-b811-0e9de3866d37" containerName="revoke-keys"
Nov 02 15:10:48.353867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353398    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c163f924-dc2f-4416-8d58-a5e0319dff6e" containerName="revoke-keys"
Nov 02 15:10:48.353867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.353411    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" containerName="revoke-keys"
Nov 02 15:10:48.364536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.364514    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"]
Nov 02 15:10:48.364761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.364744    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.382545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.382520    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:10:48.382844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.382737    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:10:48.465768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.465729    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:10:48.467850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.467829    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:10:48.533178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.533109    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.533178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.533156    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7dj\" (UniqueName: \"kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.533643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.533258    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.634593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.634553    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.634593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.634638    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7dj\" (UniqueName: \"kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.634593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.634704    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.635674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.635646    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.638031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.638000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.656851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.656823    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7dj\" (UniqueName: \"kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj\") pod \"maas-api-revoke-sub-79ab3e53-rzcpb\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:48.690633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:48.690604    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:49.066513 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:49.064891    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8b124f_bc48_4e66_83b9_d15bd525d842.slice/crio-06f8ecf2a9bba996272423b2a08bc9f870e8ffc0de169a399eadc3bad7f62829 WatchSource:0}: Error finding container 06f8ecf2a9bba996272423b2a08bc9f870e8ffc0de169a399eadc3bad7f62829: Status 404 returned error can't find the container with id 06f8ecf2a9bba996272423b2a08bc9f870e8ffc0de169a399eadc3bad7f62829
Nov 02 15:10:49.070549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.068014    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"]
Nov 02 15:10:49.271174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.271144    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:49.271979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.271955    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:49.272687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.272666    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:10:49.826895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.826784    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb" event={"ID":"5e8b124f-bc48-4e66-83b9-d15bd525d842","Type":"ContainerStarted","Data":"8b5d3fd2c4c7b08b292ced3842482f5c8be8fb54cf2fd8557fb9094c6117a50b"}
Nov 02 15:10:49.826895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.826849    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb" event={"ID":"5e8b124f-bc48-4e66-83b9-d15bd525d842","Type":"ContainerStarted","Data":"06f8ecf2a9bba996272423b2a08bc9f870e8ffc0de169a399eadc3bad7f62829"}
Nov 02 15:10:49.864929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:49.864879    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb" podStartSLOduration=1.864867061 podStartE2EDuration="1.864867061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:49.863215562 +0000 UTC m=+902.121285011" watchObservedRunningTime="2026-11-02 15:10:49.864867061 +0000 UTC m=+902.122936510"
Nov 02 15:10:50.840619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:50.840587    5041 generic.go:356] "Generic (PLEG): container finished" podID="5e8b124f-bc48-4e66-83b9-d15bd525d842" containerID="8b5d3fd2c4c7b08b292ced3842482f5c8be8fb54cf2fd8557fb9094c6117a50b" exitCode=0
Nov 02 15:10:50.841008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:50.840661    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb" event={"ID":"5e8b124f-bc48-4e66-83b9-d15bd525d842","Type":"ContainerDied","Data":"8b5d3fd2c4c7b08b292ced3842482f5c8be8fb54cf2fd8557fb9094c6117a50b"}
Nov 02 15:10:51.190137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.190100    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"]
Nov 02 15:10:51.194701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.194678    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.213358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.213335    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"distinct-2-w3-624254-kserve-self-signed-certs\""
Nov 02 15:10:51.341690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.341649    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"]
Nov 02 15:10:51.380994 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.380957    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcqss\" (UniqueName: \"kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.381147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.381060    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.381147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.381091    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.381147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.381108    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.381262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.381244    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.381303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.381287    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.484910    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqss\" (UniqueName: \"kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.485009    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.485062    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.485085    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.485127    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.485164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.485868    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.487429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.486592    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.499389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.498355    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.553390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.544132    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqss\" (UniqueName: \"kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.583445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.567945    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.584298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.584268    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs\") pod \"distinct-2-w3-624254-kserve-8c69cdcd-6nnnv\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") " pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:51.809113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:51.809028    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:52.051054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.050908    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:52.159114 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:52.159082    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd99d03cc_23a7_4c93_ab72_9d1e59372cd5.slice/crio-53de9d5714f73c0f14022d27cdafd89a4955f8a8414465ee3fa964e370a595c1 WatchSource:0}: Error finding container 53de9d5714f73c0f14022d27cdafd89a4955f8a8414465ee3fa964e370a595c1: Status 404 returned error can't find the container with id 53de9d5714f73c0f14022d27cdafd89a4955f8a8414465ee3fa964e370a595c1
Nov 02 15:10:52.160863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.160832    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"]
Nov 02 15:10:52.228464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.228441    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca\") pod \"5e8b124f-bc48-4e66-83b9-d15bd525d842\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") "
Nov 02 15:10:52.228608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.228589    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj\" (UniqueName: \"kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj\") pod \"5e8b124f-bc48-4e66-83b9-d15bd525d842\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") "
Nov 02 15:10:52.228682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.228641    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca\") pod \"5e8b124f-bc48-4e66-83b9-d15bd525d842\" (UID: \"5e8b124f-bc48-4e66-83b9-d15bd525d842\") "
Nov 02 15:10:52.229249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.229031    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca" pod "5e8b124f-bc48-4e66-83b9-d15bd525d842" (UID: "5e8b124f-bc48-4e66-83b9-d15bd525d842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:52.229249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.229220    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:52.230651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.230628    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca" pod "5e8b124f-bc48-4e66-83b9-d15bd525d842" (UID: "5e8b124f-bc48-4e66-83b9-d15bd525d842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:52.230651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.230643    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj" pod "5e8b124f-bc48-4e66-83b9-d15bd525d842" (UID: "5e8b124f-bc48-4e66-83b9-d15bd525d842"). InnerVolumeSpecName "kube-api-access-2g7dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:52.330326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.330292    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7dj\" (UniqueName: \"kubernetes.io/projected/5e8b124f-bc48-4e66-83b9-d15bd525d842-kube-api-access-2g7dj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:52.330326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.330318    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8b124f-bc48-4e66-83b9-d15bd525d842-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:52.861148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.861078    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"
Nov 02 15:10:52.861287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.861079    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb" event={"ID":"5e8b124f-bc48-4e66-83b9-d15bd525d842","Type":"ContainerDied","Data":"06f8ecf2a9bba996272423b2a08bc9f870e8ffc0de169a399eadc3bad7f62829"}
Nov 02 15:10:52.861287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.861216    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f8ecf2a9bba996272423b2a08bc9f870e8ffc0de169a399eadc3bad7f62829"
Nov 02 15:10:52.863041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.863016    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" event={"ID":"d99d03cc-23a7-4c93-ab72-9d1e59372cd5","Type":"ContainerStarted","Data":"29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653"}
Nov 02 15:10:52.863171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.863049    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" event={"ID":"d99d03cc-23a7-4c93-ab72-9d1e59372cd5","Type":"ContainerStarted","Data":"53de9d5714f73c0f14022d27cdafd89a4955f8a8414465ee3fa964e370a595c1"}
Nov 02 15:10:52.904072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:52.904019    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podStartSLOduration=1.904007803 podStartE2EDuration="1.904007803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:52.899163608 +0000 UTC m=+905.157233059" watchObservedRunningTime="2026-11-02 15:10:52.904007803 +0000 UTC m=+905.162077253"
Nov 02 15:10:53.184606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.184568    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"]
Nov 02 15:10:53.185300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.185282    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5e8b124f-bc48-4e66-83b9-d15bd525d842" containerName="revoke-keys"
Nov 02 15:10:53.185354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.185302    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8b124f-bc48-4e66-83b9-d15bd525d842" containerName="revoke-keys"
Nov 02 15:10:53.185479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.185468    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8b124f-bc48-4e66-83b9-d15bd525d842" containerName="revoke-keys"
Nov 02 15:10:53.189327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.189310    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.204375 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.204341    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:10:53.204631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.204614    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:10:53.210727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.210685    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"]
Nov 02 15:10:53.348825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.348796    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghs9\" (UniqueName: \"kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.348960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.348911    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.349009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.348990    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.450149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.450074    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.450270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.450192    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghs9\" (UniqueName: \"kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.450312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.450276    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.450730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.450705    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.452858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.452830    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.470951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.470927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghs9\" (UniqueName: \"kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9\") pod \"maas-api-revoke-sub-5db3d36a-99vl9\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.503493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.503473    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:53.809738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.809665    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:10:53.813865 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:53.813834    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03079dc_4006_4658_a57d_cfe73dc2e567.slice/crio-fee146d4197056e870d74a540ced07e46a65fef3cfa6705446ea00e2e3aa90f3 WatchSource:0}: Error finding container fee146d4197056e870d74a540ced07e46a65fef3cfa6705446ea00e2e3aa90f3: Status 404 returned error can't find the container with id fee146d4197056e870d74a540ced07e46a65fef3cfa6705446ea00e2e3aa90f3
Nov 02 15:10:53.814944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.814923    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"]
Nov 02 15:10:53.872502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:53.872477    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9" event={"ID":"d03079dc-4006-4658-a57d-cfe73dc2e567","Type":"ContainerStarted","Data":"fee146d4197056e870d74a540ced07e46a65fef3cfa6705446ea00e2e3aa90f3"}
Nov 02 15:10:54.882470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:54.882439    5041 generic.go:356] "Generic (PLEG): container finished" podID="d03079dc-4006-4658-a57d-cfe73dc2e567" containerID="f0a530bd35e39e7e5c345aae3dac896efb771eed8d87fff2c141af86a758c9ba" exitCode=0
Nov 02 15:10:54.882900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:54.882474    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9" event={"ID":"d03079dc-4006-4658-a57d-cfe73dc2e567","Type":"ContainerDied","Data":"f0a530bd35e39e7e5c345aae3dac896efb771eed8d87fff2c141af86a758c9ba"}
Nov 02 15:10:56.060294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.060268    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:56.210268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.210232    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9\" (UniqueName: \"kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9\") pod \"d03079dc-4006-4658-a57d-cfe73dc2e567\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") "
Nov 02 15:10:56.210499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.210275    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca\") pod \"d03079dc-4006-4658-a57d-cfe73dc2e567\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") "
Nov 02 15:10:56.210499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.210444    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca\") pod \"d03079dc-4006-4658-a57d-cfe73dc2e567\" (UID: \"d03079dc-4006-4658-a57d-cfe73dc2e567\") "
Nov 02 15:10:56.210808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.210772    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca" pod "d03079dc-4006-4658-a57d-cfe73dc2e567" (UID: "d03079dc-4006-4658-a57d-cfe73dc2e567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:56.211536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.211508    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:56.213112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.213079    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca" pod "d03079dc-4006-4658-a57d-cfe73dc2e567" (UID: "d03079dc-4006-4658-a57d-cfe73dc2e567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:56.213389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.213343    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9" pod "d03079dc-4006-4658-a57d-cfe73dc2e567" (UID: "d03079dc-4006-4658-a57d-cfe73dc2e567"). InnerVolumeSpecName "kube-api-access-wghs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:56.312467 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.312430    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghs9\" (UniqueName: \"kubernetes.io/projected/d03079dc-4006-4658-a57d-cfe73dc2e567-kube-api-access-wghs9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:56.312467 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.312465    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03079dc-4006-4658-a57d-cfe73dc2e567-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:10:56.901142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.901114    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"
Nov 02 15:10:56.901142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.901120    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9" event={"ID":"d03079dc-4006-4658-a57d-cfe73dc2e567","Type":"ContainerDied","Data":"fee146d4197056e870d74a540ced07e46a65fef3cfa6705446ea00e2e3aa90f3"}
Nov 02 15:10:56.901348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:56.901165    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee146d4197056e870d74a540ced07e46a65fef3cfa6705446ea00e2e3aa90f3"
Nov 02 15:10:57.992529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:57.992500    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"]
Nov 02 15:10:57.993105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:57.993092    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d03079dc-4006-4658-a57d-cfe73dc2e567" containerName="revoke-keys"
Nov 02 15:10:57.993157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:57.993106    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03079dc-4006-4658-a57d-cfe73dc2e567" containerName="revoke-keys"
Nov 02 15:10:57.993250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:57.993241    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03079dc-4006-4658-a57d-cfe73dc2e567" containerName="revoke-keys"
Nov 02 15:10:57.997139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:57.997118    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.010284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.010264    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:10:58.027420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.025747    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"]
Nov 02 15:10:58.111051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.111016    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:10:58.135263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.135233    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm79f\" (UniqueName: \"kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.135391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.135300    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.135476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.135456    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.236647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.236621    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.236756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.236694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.236806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.236769    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm79f\" (UniqueName: \"kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.237420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.237387    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.238966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.238945    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.259977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.259927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm79f\" (UniqueName: \"kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f\") pod \"maas-api-revoke-sub-3d2697c2-mbtc5\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.310894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.310875    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:10:58.628598 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:10:58.628567    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a7a3d4_889c_4316_afc2_ff3ee94e60bc.slice/crio-6716423f063d6944e9b68a96ab9853e987ca945239f415d50f41c2689dd8ffac WatchSource:0}: Error finding container 6716423f063d6944e9b68a96ab9853e987ca945239f415d50f41c2689dd8ffac: Status 404 returned error can't find the container with id 6716423f063d6944e9b68a96ab9853e987ca945239f415d50f41c2689dd8ffac
Nov 02 15:10:58.630177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.630150    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"]
Nov 02 15:10:58.770921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.770894    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"]
Nov 02 15:10:58.776689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.776670    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.796194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.796169    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"]
Nov 02 15:10:58.843755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.843721    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhgx\" (UniqueName: \"kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.843870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.843847    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.843919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.843903    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.920340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.920305    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5" event={"ID":"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc","Type":"ContainerStarted","Data":"6716423f063d6944e9b68a96ab9853e987ca945239f415d50f41c2689dd8ffac"}
Nov 02 15:10:58.944907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.944879    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.945001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.944938    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.945058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.945028    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhgx\" (UniqueName: \"kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.945592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.945570    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.947094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.947074    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:58.965969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:58.965951    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhgx\" (UniqueName: \"kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx\") pod \"maas-api-revoke-sub-afa5874c-rtlj6\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:59.090851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:59.090825    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:10:59.454386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:59.448133    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"]
Nov 02 15:10:59.931317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:59.931289    5041 generic.go:356] "Generic (PLEG): container finished" podID="f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" containerID="93ae1c5fda5d6c63332fa7c3cc957bcccdceb718fd5a860e870852974f416487" exitCode=0
Nov 02 15:10:59.931463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:59.931382    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5" event={"ID":"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc","Type":"ContainerDied","Data":"93ae1c5fda5d6c63332fa7c3cc957bcccdceb718fd5a860e870852974f416487"}
Nov 02 15:10:59.932955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:59.932930    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6" event={"ID":"7157c814-23ee-4f1d-b4e6-82723bdbd3a5","Type":"ContainerStarted","Data":"fc8af10c95f7d67278cc5c0ffb93991e78529fdf3120275851a2d3fcebdd2c24"}
Nov 02 15:10:59.933020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:10:59.932982    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6" event={"ID":"7157c814-23ee-4f1d-b4e6-82723bdbd3a5","Type":"ContainerStarted","Data":"5e86ddf7db409ff643e2bebdcc4e71b7de1b1f94a1b9e834a836f0c08f69dd38"}
Nov 02 15:11:00.006064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:00.005982    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6" podStartSLOduration=2.005971398 podStartE2EDuration="2.005971398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:00.005218172 +0000 UTC m=+912.263287621" watchObservedRunningTime="2026-11-02 15:11:00.005971398 +0000 UTC m=+912.264040847"
Nov 02 15:11:00.945288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:00.945252    5041 generic.go:356] "Generic (PLEG): container finished" podID="7157c814-23ee-4f1d-b4e6-82723bdbd3a5" containerID="fc8af10c95f7d67278cc5c0ffb93991e78529fdf3120275851a2d3fcebdd2c24" exitCode=0
Nov 02 15:11:00.945708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:00.945337    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6" event={"ID":"7157c814-23ee-4f1d-b4e6-82723bdbd3a5","Type":"ContainerDied","Data":"fc8af10c95f7d67278cc5c0ffb93991e78529fdf3120275851a2d3fcebdd2c24"}
Nov 02 15:11:01.196016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.195959    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:11:01.325012 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.324984    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f\" (UniqueName: \"kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f\") pod \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") "
Nov 02 15:11:01.325160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.325035    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca\") pod \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") "
Nov 02 15:11:01.325160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.325067    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca\") pod \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\" (UID: \"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc\") "
Nov 02 15:11:01.325536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.325505    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca" pod "f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" (UID: "f3a7a3d4-889c-4316-afc2-ff3ee94e60bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:01.325636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.325623    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:01.327111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.327089    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca" pod "f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" (UID: "f3a7a3d4-889c-4316-afc2-ff3ee94e60bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:01.327199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.327183    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f" pod "f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" (UID: "f3a7a3d4-889c-4316-afc2-ff3ee94e60bc"). InnerVolumeSpecName "kube-api-access-hm79f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:01.426707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.426682    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm79f\" (UniqueName: \"kubernetes.io/projected/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-kube-api-access-hm79f\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:01.426707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.426703    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:01.809908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.809877    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:01.822413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.822357    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:01.955747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.955721    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"
Nov 02 15:11:01.956183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.955729    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5" event={"ID":"f3a7a3d4-889c-4316-afc2-ff3ee94e60bc","Type":"ContainerDied","Data":"6716423f063d6944e9b68a96ab9853e987ca945239f415d50f41c2689dd8ffac"}
Nov 02 15:11:01.956183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.955793    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6716423f063d6944e9b68a96ab9853e987ca945239f415d50f41c2689dd8ffac"
Nov 02 15:11:01.968286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:01.968266    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:02.142179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.142147    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:11:02.236834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.236805    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca\") pod \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") "
Nov 02 15:11:02.237009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.236846    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx\" (UniqueName: \"kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx\") pod \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") "
Nov 02 15:11:02.237009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.236866    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca\") pod \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\" (UID: \"7157c814-23ee-4f1d-b4e6-82723bdbd3a5\") "
Nov 02 15:11:02.237328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.237299    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca" pod "7157c814-23ee-4f1d-b4e6-82723bdbd3a5" (UID: "7157c814-23ee-4f1d-b4e6-82723bdbd3a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:02.237614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.237595    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:02.239402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.239350    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx" pod "7157c814-23ee-4f1d-b4e6-82723bdbd3a5" (UID: "7157c814-23ee-4f1d-b4e6-82723bdbd3a5"). InnerVolumeSpecName "kube-api-access-7zhgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:02.239574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.239548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca" pod "7157c814-23ee-4f1d-b4e6-82723bdbd3a5" (UID: "7157c814-23ee-4f1d-b4e6-82723bdbd3a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:02.338777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.338738    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:02.338777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.338776    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhgx\" (UniqueName: \"kubernetes.io/projected/7157c814-23ee-4f1d-b4e6-82723bdbd3a5-kube-api-access-7zhgx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:02.965699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.965666    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"
Nov 02 15:11:02.966091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.965697    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6" event={"ID":"7157c814-23ee-4f1d-b4e6-82723bdbd3a5","Type":"ContainerDied","Data":"5e86ddf7db409ff643e2bebdcc4e71b7de1b1f94a1b9e834a836f0c08f69dd38"}
Nov 02 15:11:02.966091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:02.965751    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e86ddf7db409ff643e2bebdcc4e71b7de1b1f94a1b9e834a836f0c08f69dd38"
Nov 02 15:11:03.651061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.651027    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"]
Nov 02 15:11:03.651892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.651867    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" containerName="revoke-keys"
Nov 02 15:11:03.651892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.651889    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" containerName="revoke-keys"
Nov 02 15:11:03.652076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.651933    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7157c814-23ee-4f1d-b4e6-82723bdbd3a5" containerName="revoke-keys"
Nov 02 15:11:03.652076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.651939    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7157c814-23ee-4f1d-b4e6-82723bdbd3a5" containerName="revoke-keys"
Nov 02 15:11:03.652076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.652050    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7157c814-23ee-4f1d-b4e6-82723bdbd3a5" containerName="revoke-keys"
Nov 02 15:11:03.652076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.652071    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" containerName="revoke-keys"
Nov 02 15:11:03.656226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.656200    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.683923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.683897    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"]
Nov 02 15:11:03.758435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758405    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758436    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758462    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758609    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbpt\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758664    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758682    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758762    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758810    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.758927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.758916    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860241    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860275    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860292    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860352    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbpt\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860446    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860476    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860546    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860623    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860688    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860812    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.860944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.861007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.860977    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.861146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.861122    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.861245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.861226    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.862667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.862646    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.862879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.862862    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.886853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.886824    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.887170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.887148    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbpt\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:03.975597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:03.975573    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:04.295956 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:04.295928    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9947727b_4881_419f_a606_55bc58437a3c.slice/crio-d80ce163728c3bfe79fe843c1af5cd89c43edb6fac35d26664d079cccfcfc3fb WatchSource:0}: Error finding container d80ce163728c3bfe79fe843c1af5cd89c43edb6fac35d26664d079cccfcfc3fb: Status 404 returned error can't find the container with id d80ce163728c3bfe79fe843c1af5cd89c43edb6fac35d26664d079cccfcfc3fb
Nov 02 15:11:04.298616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:04.298595    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"]
Nov 02 15:11:04.299884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:04.299847    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:04.299952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:04.299908    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:04.299952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:04.299939    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:04.983775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:04.983740    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" event={"ID":"9947727b-4881-419f-a606-55bc58437a3c","Type":"ContainerStarted","Data":"0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f"}
Nov 02 15:11:04.984194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:04.983791    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" event={"ID":"9947727b-4881-419f-a606-55bc58437a3c","Type":"ContainerStarted","Data":"d80ce163728c3bfe79fe843c1af5cd89c43edb6fac35d26664d079cccfcfc3fb"}
Nov 02 15:11:05.047348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:05.047289    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podStartSLOduration=2.047274778 podStartE2EDuration="2.047274778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:05.035902713 +0000 UTC m=+917.293972165" watchObservedRunningTime="2026-11-02 15:11:05.047274778 +0000 UTC m=+917.305344227"
Nov 02 15:11:05.996387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:05.983599    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:05.996387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:05.983737    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:05.996387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:05.983780    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:06.835393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.834182    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:06.897396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.894434    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:06.912350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.911620    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:11:06.913101 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:11:06.913038    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:11:06.950982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.950457    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:06.977871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.977840    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:06.978116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.978089    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:06.994009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:06.993972    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.288637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.287447    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.358498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.358453    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.730389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.729459    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.749385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.746687    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.836385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.834850    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.836385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.835283    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:11:07.895899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.894720    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.920397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:07.912661    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.050581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.036704    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:08.050581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.036761    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:08.050581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.037139    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.050581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.037193    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.050581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.038387    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.321681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.321580    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.548391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.546627    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.563389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.556474    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.709387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.681478    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:11:08.709387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.681602    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:11:08.709387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.681744    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:11:08.709387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.681784    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:08.709387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.681822    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:11:08.709387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.681861    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:11:08.749477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.749435    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.766550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.766517    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:11:08.799397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.794539    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:08.933228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.906064    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.933228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.924047    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:11:08.933228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.924595    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.933228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:08.928598    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:11:09.014373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.012910    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:11:09.014373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.013022    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:09.014373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.013070    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:09.390440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.390065    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.806521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.792660    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.810960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.806484    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:11:09.980696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.930454    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 15:11:09.980696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.978068    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.980958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.980782    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:09.980958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:09.980830    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:10.113687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.113584    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.177040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.176995    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.247420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.230160    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:11:10.251476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.249824    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:11:10.288583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.288552    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:11:10.294438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.291340    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:11:10.297816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.297621    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:11:10.977777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.977738    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:10.979464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:10.979428    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:11.976049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:11.976015    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:11.976218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:11.976069    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:12.040266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.040230    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"]
Nov 02 15:11:12.045575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.045551    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.063815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.063785    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"]
Nov 02 15:11:12.064265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.064245    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:12.064402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.064328    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:12.178998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.178961    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.178998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.178997    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6rt\" (UniqueName: \"kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.179186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.179169    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.280665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.280586    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.280844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.280675    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.280844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.280697    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6rt\" (UniqueName: \"kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.281334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.281312    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.282950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.282925    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.301712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.301677    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6rt\" (UniqueName: \"kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt\") pod \"maas-api-revoke-sub-dea35dc8-zn72l\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.363277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.363229    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:12.695140 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:12.695107    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9011bd_2757_48d8_87dc_d37e650a17ef.slice/crio-aee69ad35a08c6fcecf6ea2846165f3f5ea043439ac95b987591e2da102ded6d WatchSource:0}: Error finding container aee69ad35a08c6fcecf6ea2846165f3f5ea043439ac95b987591e2da102ded6d: Status 404 returned error can't find the container with id aee69ad35a08c6fcecf6ea2846165f3f5ea043439ac95b987591e2da102ded6d
Nov 02 15:11:12.699970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.699925    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"]
Nov 02 15:11:12.854576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.854534    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l" event={"ID":"5e9011bd-2757-48d8-87dc-d37e650a17ef","Type":"ContainerStarted","Data":"aee69ad35a08c6fcecf6ea2846165f3f5ea043439ac95b987591e2da102ded6d"}
Nov 02 15:11:12.976528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.976458    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 15:11:12.976528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:12.976511    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 15:11:13.866133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:13.866098    5041 generic.go:356] "Generic (PLEG): container finished" podID="5e9011bd-2757-48d8-87dc-d37e650a17ef" containerID="2b9a6b3a644bead7985328fddb98134c6ebb0d9fb8c067a651f583304fc52a7a" exitCode=0
Nov 02 15:11:13.866546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:13.866184    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l" event={"ID":"5e9011bd-2757-48d8-87dc-d37e650a17ef","Type":"ContainerDied","Data":"2b9a6b3a644bead7985328fddb98134c6ebb0d9fb8c067a651f583304fc52a7a"}
Nov 02 15:11:14.059243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.059207    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"]
Nov 02 15:11:14.065578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.065553    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.085100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.085068    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"]
Nov 02 15:11:14.211202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.211150    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj58t\" (UniqueName: \"kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.211491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.211452    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.211621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.211501    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.312639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.312542    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.312639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.312603    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.312854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.312686    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj58t\" (UniqueName: \"kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.313506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.313474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.315180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.315155    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.334079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.334055    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj58t\" (UniqueName: \"kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t\") pod \"maas-api-revoke-sub-da75a42e-xskvz\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.380270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.380248    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:14.804685 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:14.804653    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3f883e_4e20_4f33_bd8c_3cef098d0a84.slice/crio-19439b0c9ba28ce23b6e2f2191c786f937b52c9b3693bc4453b507104150b6a6 WatchSource:0}: Error finding container 19439b0c9ba28ce23b6e2f2191c786f937b52c9b3693bc4453b507104150b6a6: Status 404 returned error can't find the container with id 19439b0c9ba28ce23b6e2f2191c786f937b52c9b3693bc4453b507104150b6a6
Nov 02 15:11:14.805409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.805344    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"]
Nov 02 15:11:14.879192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.879162    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz" event={"ID":"3b3f883e-4e20-4f33-bd8c-3cef098d0a84","Type":"ContainerStarted","Data":"19439b0c9ba28ce23b6e2f2191c786f937b52c9b3693bc4453b507104150b6a6"}
Nov 02 15:11:14.972328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.970144    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:14.972328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.971447    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:14.974099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:14.972943    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:11:15.034465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.034437    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:15.225899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.225869    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca\") pod \"5e9011bd-2757-48d8-87dc-d37e650a17ef\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") "
Nov 02 15:11:15.226099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.226030    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt\" (UniqueName: \"kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt\") pod \"5e9011bd-2757-48d8-87dc-d37e650a17ef\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") "
Nov 02 15:11:15.226206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.226183    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca\") pod \"5e9011bd-2757-48d8-87dc-d37e650a17ef\" (UID: \"5e9011bd-2757-48d8-87dc-d37e650a17ef\") "
Nov 02 15:11:15.226328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.226231    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca" pod "5e9011bd-2757-48d8-87dc-d37e650a17ef" (UID: "5e9011bd-2757-48d8-87dc-d37e650a17ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:15.226759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.226737    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:15.228146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.228126    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt" pod "5e9011bd-2757-48d8-87dc-d37e650a17ef" (UID: "5e9011bd-2757-48d8-87dc-d37e650a17ef"). InnerVolumeSpecName "kube-api-access-cf6rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:15.228745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.228720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca" pod "5e9011bd-2757-48d8-87dc-d37e650a17ef" (UID: "5e9011bd-2757-48d8-87dc-d37e650a17ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:15.328562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.328530    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6rt\" (UniqueName: \"kubernetes.io/projected/5e9011bd-2757-48d8-87dc-d37e650a17ef-kube-api-access-cf6rt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:15.328562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.328553    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9011bd-2757-48d8-87dc-d37e650a17ef-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:15.891995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.891967    5041 generic.go:356] "Generic (PLEG): container finished" podID="3b3f883e-4e20-4f33-bd8c-3cef098d0a84" containerID="0d130197fd6ad75d94c9c616fb1ccc4d25b106f2a25aa4692e8c8bae2603ea3a" exitCode=0
Nov 02 15:11:15.892438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.892042    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz" event={"ID":"3b3f883e-4e20-4f33-bd8c-3cef098d0a84","Type":"ContainerDied","Data":"0d130197fd6ad75d94c9c616fb1ccc4d25b106f2a25aa4692e8c8bae2603ea3a"}
Nov 02 15:11:15.893666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.893647    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"
Nov 02 15:11:15.893800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.893707    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l" event={"ID":"5e9011bd-2757-48d8-87dc-d37e650a17ef","Type":"ContainerDied","Data":"aee69ad35a08c6fcecf6ea2846165f3f5ea043439ac95b987591e2da102ded6d"}
Nov 02 15:11:15.893800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:15.893757    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee69ad35a08c6fcecf6ea2846165f3f5ea043439ac95b987591e2da102ded6d"
Nov 02 15:11:17.087770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.087734    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:17.250998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.250962    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t\" (UniqueName: \"kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t\") pod \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") "
Nov 02 15:11:17.251191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.251021    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca\") pod \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") "
Nov 02 15:11:17.251191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.251068    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca\") pod \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\" (UID: \"3b3f883e-4e20-4f33-bd8c-3cef098d0a84\") "
Nov 02 15:11:17.251537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.251508    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca" pod "3b3f883e-4e20-4f33-bd8c-3cef098d0a84" (UID: "3b3f883e-4e20-4f33-bd8c-3cef098d0a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:17.251727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.251705    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:17.253437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.253405    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca" pod "3b3f883e-4e20-4f33-bd8c-3cef098d0a84" (UID: "3b3f883e-4e20-4f33-bd8c-3cef098d0a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:17.253645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.253621    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t" pod "3b3f883e-4e20-4f33-bd8c-3cef098d0a84" (UID: "3b3f883e-4e20-4f33-bd8c-3cef098d0a84"). InnerVolumeSpecName "kube-api-access-nj58t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:17.353191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.353163    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj58t\" (UniqueName: \"kubernetes.io/projected/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-kube-api-access-nj58t\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:17.353191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.353190    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b3f883e-4e20-4f33-bd8c-3cef098d0a84-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:17.932913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.932882    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"
Nov 02 15:11:17.933091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.932885    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz" event={"ID":"3b3f883e-4e20-4f33-bd8c-3cef098d0a84","Type":"ContainerDied","Data":"19439b0c9ba28ce23b6e2f2191c786f937b52c9b3693bc4453b507104150b6a6"}
Nov 02 15:11:17.933091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:17.933007    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19439b0c9ba28ce23b6e2f2191c786f937b52c9b3693bc4453b507104150b6a6"
Nov 02 15:11:18.389762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.389681    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"]
Nov 02 15:11:18.390403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.390384    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5e9011bd-2757-48d8-87dc-d37e650a17ef" containerName="revoke-keys"
Nov 02 15:11:18.390403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.390401    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9011bd-2757-48d8-87dc-d37e650a17ef" containerName="revoke-keys"
Nov 02 15:11:18.390528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.390411    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3b3f883e-4e20-4f33-bd8c-3cef098d0a84" containerName="revoke-keys"
Nov 02 15:11:18.390528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.390416    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3f883e-4e20-4f33-bd8c-3cef098d0a84" containerName="revoke-keys"
Nov 02 15:11:18.390528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.390518    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9011bd-2757-48d8-87dc-d37e650a17ef" containerName="revoke-keys"
Nov 02 15:11:18.390616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.390529    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3f883e-4e20-4f33-bd8c-3cef098d0a84" containerName="revoke-keys"
Nov 02 15:11:18.394459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.394441    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.408986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.408966    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:18.409098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.409082    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:18.416047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.416021    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"]
Nov 02 15:11:18.570111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.570083    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.570291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.570118    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjrp\" (UniqueName: \"kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.570291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.570241    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.671765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.671731    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.671922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.671773    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjrp\" (UniqueName: \"kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.671922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.671816    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.672452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.672430    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.674190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.674170    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.694949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.694923    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjrp\" (UniqueName: \"kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp\") pod \"maas-api-revoke-sub-e80a3e2a-ztj48\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:18.708764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:18.708742    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:19.020150 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:19.020119    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4253f7d7_59f3_462a_884e_1a91303fd2f1.slice/crio-aaf3fc968d77085e3930c2f9823ec577d6039374d79dc48a07ff7f0f5c18f76d WatchSource:0}: Error finding container aaf3fc968d77085e3930c2f9823ec577d6039374d79dc48a07ff7f0f5c18f76d: Status 404 returned error can't find the container with id aaf3fc968d77085e3930c2f9823ec577d6039374d79dc48a07ff7f0f5c18f76d
Nov 02 15:11:19.022911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:19.022825    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"]
Nov 02 15:11:19.952650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:19.952618    5041 generic.go:356] "Generic (PLEG): container finished" podID="4253f7d7-59f3-462a-884e-1a91303fd2f1" containerID="27c302a2945feff4086c3e25af8c3746958635971a0e62236f6a22d3d96687f9" exitCode=0
Nov 02 15:11:19.952967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:19.952692    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48" event={"ID":"4253f7d7-59f3-462a-884e-1a91303fd2f1","Type":"ContainerDied","Data":"27c302a2945feff4086c3e25af8c3746958635971a0e62236f6a22d3d96687f9"}
Nov 02 15:11:19.952967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:19.952746    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48" event={"ID":"4253f7d7-59f3-462a-884e-1a91303fd2f1","Type":"ContainerStarted","Data":"aaf3fc968d77085e3930c2f9823ec577d6039374d79dc48a07ff7f0f5c18f76d"}
Nov 02 15:11:21.115794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.115769    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:21.203918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.203887    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca\") pod \"4253f7d7-59f3-462a-884e-1a91303fd2f1\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") "
Nov 02 15:11:21.204095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.204052    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca\") pod \"4253f7d7-59f3-462a-884e-1a91303fd2f1\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") "
Nov 02 15:11:21.204188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.204160    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp\" (UniqueName: \"kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp\") pod \"4253f7d7-59f3-462a-884e-1a91303fd2f1\" (UID: \"4253f7d7-59f3-462a-884e-1a91303fd2f1\") "
Nov 02 15:11:21.204255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.204232    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca" pod "4253f7d7-59f3-462a-884e-1a91303fd2f1" (UID: "4253f7d7-59f3-462a-884e-1a91303fd2f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:21.204770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.204746    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:21.206118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.206090    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca" pod "4253f7d7-59f3-462a-884e-1a91303fd2f1" (UID: "4253f7d7-59f3-462a-884e-1a91303fd2f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:21.206217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.206150    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp" pod "4253f7d7-59f3-462a-884e-1a91303fd2f1" (UID: "4253f7d7-59f3-462a-884e-1a91303fd2f1"). InnerVolumeSpecName "kube-api-access-vsjrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:21.307402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.306287    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253f7d7-59f3-462a-884e-1a91303fd2f1-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:21.307402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.306460    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjrp\" (UniqueName: \"kubernetes.io/projected/4253f7d7-59f3-462a-884e-1a91303fd2f1-kube-api-access-vsjrp\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:21.971357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.971318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48" event={"ID":"4253f7d7-59f3-462a-884e-1a91303fd2f1","Type":"ContainerDied","Data":"aaf3fc968d77085e3930c2f9823ec577d6039374d79dc48a07ff7f0f5c18f76d"}
Nov 02 15:11:21.971357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.971385    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf3fc968d77085e3930c2f9823ec577d6039374d79dc48a07ff7f0f5c18f76d"
Nov 02 15:11:21.971560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:21.971334    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"
Nov 02 15:11:23.905053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.905021    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"]
Nov 02 15:11:23.905677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.905661    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4253f7d7-59f3-462a-884e-1a91303fd2f1" containerName="revoke-keys"
Nov 02 15:11:23.905677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.905677    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4253f7d7-59f3-462a-884e-1a91303fd2f1" containerName="revoke-keys"
Nov 02 15:11:23.905935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.905826    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4253f7d7-59f3-462a-884e-1a91303fd2f1" containerName="revoke-keys"
Nov 02 15:11:23.909913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.909893    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941206    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941306    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941351    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941401    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5x7l\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941493    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941542    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941605    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941631    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.941921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.941696    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:23.944855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:23.944830    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"]
Nov 02 15:11:24.043131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043092    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043145    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043176    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5x7l\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043261    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043307    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043412    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043440    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043589    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043600    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.043954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.043952    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.044150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.044129    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.044221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.044145    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.046015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.045989    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.046118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.046055    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.064767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.064747    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.064935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.064907    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5x7l\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.202946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.202916    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"]
Nov 02 15:11:24.210268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.210247    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.225771 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.225747    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:24.226343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.226322    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:24.226564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.226536    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:24.230070 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.230040    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"]
Nov 02 15:11:24.251831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.251486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.251831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.251584    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gp9\" (UniqueName: \"kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.251995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.251837    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.354391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.353489    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.354391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.353565    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.354391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.353640    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gp9\" (UniqueName: \"kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.354913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.354858    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.356889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.356860    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.374742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.374722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gp9\" (UniqueName: \"kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9\") pod \"maas-api-revoke-sub-a7d7a50e-b2kpb\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.469435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.468311    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"]
Nov 02 15:11:24.516596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.516556    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.529346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.525964    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:24.537676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.534025    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"]
Nov 02 15:11:24.557446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557345    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557440    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557507    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557527    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557595    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557731    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557755    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557775    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.557879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.557846    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k96cz\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.601662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.601337    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"]
Nov 02 15:11:24.603999 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:24.602756    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f96ca2_28a7_4268_9769_cb798ca68c87.slice/crio-4d4c7f9044d4b09111f71e831899b66cbd85439bb26db2796fc7738974e3dd04 WatchSource:0}: Error finding container 4d4c7f9044d4b09111f71e831899b66cbd85439bb26db2796fc7738974e3dd04: Status 404 returned error can't find the container with id 4d4c7f9044d4b09111f71e831899b66cbd85439bb26db2796fc7738974e3dd04
Nov 02 15:11:24.610947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.610005    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.610947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.610086    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.610947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.610133    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.659201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.659163    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.659564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.659538    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.659715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.659552    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.659846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.659828    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.659967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.659951    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660140    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k96cz\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660275    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660284    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660352    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660459    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660490    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.660776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.660748    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.661155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.661136    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.661857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.661832    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.663533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.663509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.663782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.663719    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.683463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.683445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.683925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.683899    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k96cz\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.731537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.731464    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"]
Nov 02 15:11:24.737182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.737161    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.781898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.781847    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"]
Nov 02 15:11:24.846838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.846808    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:24.861822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.861791    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"]
Nov 02 15:11:24.862231 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:24.862204    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ec1ef6_d87b_4a4a_af8a_1b9b26aff134.slice/crio-31f009311ad0af3f0a689e9cb287dc0dcbc15b648a1da4a4b5ccc38380420237 WatchSource:0}: Error finding container 31f009311ad0af3f0a689e9cb287dc0dcbc15b648a1da4a4b5ccc38380420237: Status 404 returned error can't find the container with id 31f009311ad0af3f0a689e9cb287dc0dcbc15b648a1da4a4b5ccc38380420237
Nov 02 15:11:24.864211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864178    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864298    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2wk\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864434    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864533    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864617    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864687    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864761    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.864957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864939    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.865037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.864977    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.966389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.966208    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2wk\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.966701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.966633    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.967073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.967051    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.967594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.967572    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.967702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.967683    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.968083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.968061    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.968195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.968176    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.968604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.968557    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.968860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.968839    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.969038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.969020    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.969574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.969169    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.969574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.969420    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.969869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.969850    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:24.970877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:24.970759    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:25.008218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.007735    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" event={"ID":"84f96ca2-28a7-4268-9769-cb798ca68c87","Type":"ContainerStarted","Data":"87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8"}
Nov 02 15:11:25.008218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.007805    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" event={"ID":"84f96ca2-28a7-4268-9769-cb798ca68c87","Type":"ContainerStarted","Data":"4d4c7f9044d4b09111f71e831899b66cbd85439bb26db2796fc7738974e3dd04"}
Nov 02 15:11:25.010796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.010764    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb" event={"ID":"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134","Type":"ContainerStarted","Data":"31f009311ad0af3f0a689e9cb287dc0dcbc15b648a1da4a4b5ccc38380420237"}
Nov 02 15:11:25.015758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.015726    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:25.015853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.015839    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:25.015917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.015874    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:25.016529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.016505    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2wk\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:25.054061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.054029    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:25.106523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.106456    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podStartSLOduration=2.106437316 podStartE2EDuration="2.106437316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.105144761 +0000 UTC m=+937.363214211" watchObservedRunningTime="2026-11-02 15:11:25.106437316 +0000 UTC m=+937.364506768"
Nov 02 15:11:25.215702 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:25.215631    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a6879a_f491_4692_82c5_6474fafae08d.slice/crio-d8e8a1fd1b3b7ec9eb1409383e38cb6492f68019163216f4b3252006edc130ae WatchSource:0}: Error finding container d8e8a1fd1b3b7ec9eb1409383e38cb6492f68019163216f4b3252006edc130ae: Status 404 returned error can't find the container with id d8e8a1fd1b3b7ec9eb1409383e38cb6492f68019163216f4b3252006edc130ae
Nov 02 15:11:25.219916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.219349    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.219916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.219456    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.219916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.219507    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.230419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.228504    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:25.230419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.228664    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:25.230419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.228707    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:25.251473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.250706    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"]
Nov 02 15:11:25.403548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.403521    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"]
Nov 02 15:11:25.408288 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:25.408258    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8066585_1ef2_4e74_bc7b_d90e4b3c8cbe.slice/crio-a24d29319b578f376e67a433645923255812ad7055922aadd67fad3acc51638b WatchSource:0}: Error finding container a24d29319b578f376e67a433645923255812ad7055922aadd67fad3acc51638b: Status 404 returned error can't find the container with id a24d29319b578f376e67a433645923255812ad7055922aadd67fad3acc51638b
Nov 02 15:11:25.410922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.410878    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.411109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.411087    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.411224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:25.411209    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:26.077391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.066224    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" event={"ID":"03a6879a-f491-4692-82c5-6474fafae08d","Type":"ContainerStarted","Data":"756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36"}
Nov 02 15:11:26.077391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.066321    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" event={"ID":"03a6879a-f491-4692-82c5-6474fafae08d","Type":"ContainerStarted","Data":"d8e8a1fd1b3b7ec9eb1409383e38cb6492f68019163216f4b3252006edc130ae"}
Nov 02 15:11:26.088406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.088351    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" event={"ID":"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe","Type":"ContainerStarted","Data":"924f51f5290810cf1799b09193959eddb5e6b1f7b05860938de64223f01e7b3c"}
Nov 02 15:11:26.088629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.088612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" event={"ID":"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe","Type":"ContainerStarted","Data":"a24d29319b578f376e67a433645923255812ad7055922aadd67fad3acc51638b"}
Nov 02 15:11:26.091901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.091875    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb" event={"ID":"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134","Type":"ContainerStarted","Data":"8fab1f42347e4c20d5dd72325ad86d743e235842cc5644b479280f046df051d1"}
Nov 02 15:11:26.174691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.174636    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podStartSLOduration=2.174621804 podStartE2EDuration="2.174621804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:26.137724821 +0000 UTC m=+938.395794271" watchObservedRunningTime="2026-11-02 15:11:26.174621804 +0000 UTC m=+938.432691330"
Nov 02 15:11:26.218071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.218000    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb" podStartSLOduration=2.217979385 podStartE2EDuration="2.217979385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:26.206406848 +0000 UTC m=+938.464476300" watchObservedRunningTime="2026-11-02 15:11:26.217979385 +0000 UTC m=+938.476048836"
Nov 02 15:11:26.230823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.230784    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.231093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.231065    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:26.297479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.291836    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"]
Nov 02 15:11:26.301384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.299609    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.349567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.349529    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"]
Nov 02 15:11:26.398577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.398098    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podStartSLOduration=2.398075833 podStartE2EDuration="2.398075833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:26.265987397 +0000 UTC m=+938.524056848" watchObservedRunningTime="2026-11-02 15:11:26.398075833 +0000 UTC m=+938.656145284"
Nov 02 15:11:26.480703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.480651    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5j4w\" (UniqueName: \"kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.501410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.493606    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.545183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.545144    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.738386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.713181    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.738386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.713252    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.738386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.713572    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5j4w\" (UniqueName: \"kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.738386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.728948    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.783394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.782510    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.854389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.853526    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:26.893401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.892432    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5j4w\" (UniqueName: \"kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w\") pod \"maas-api-revoke-sub-b1f3801e-ktztk\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:26.893401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.893143    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.893401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:26.893191    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:27.055444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.054783    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:27.063387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.059629    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.063387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.059697    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:27.103906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.103872    5041 generic.go:356] "Generic (PLEG): container finished" podID="48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" containerID="8fab1f42347e4c20d5dd72325ad86d743e235842cc5644b479280f046df051d1" exitCode=0
Nov 02 15:11:27.104309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.103960    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb" event={"ID":"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134","Type":"ContainerDied","Data":"8fab1f42347e4c20d5dd72325ad86d743e235842cc5644b479280f046df051d1"}
Nov 02 15:11:27.191433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.182616    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:27.271431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.269658    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.271431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.269721    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:27.706167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.706136    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"]
Nov 02 15:11:27.849058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.848634    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.849058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:27.848696    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:28.071779 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.070819    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.071779 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.070875    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:28.225666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.225623    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk" event={"ID":"addee6de-7eb8-4939-a5f2-966830e970f8","Type":"ContainerStarted","Data":"1354319db82669785730cd3185dbdeb747d12470ccf99621ea871fd390638445"}
Nov 02 15:11:28.277638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.277602    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.278614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.278582    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:28.882400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.853860    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.882816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:28.882773    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:29.133893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.133807    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.138403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.134105    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:29.267396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.266560    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.267396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.266623    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:29.407306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.406223    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:29.409666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.409627    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb" event={"ID":"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134","Type":"ContainerDied","Data":"31f009311ad0af3f0a689e9cb287dc0dcbc15b648a1da4a4b5ccc38380420237"}
Nov 02 15:11:29.410321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.410298    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31f009311ad0af3f0a689e9cb287dc0dcbc15b648a1da4a4b5ccc38380420237"
Nov 02 15:11:29.411816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.411793    5041 generic.go:356] "Generic (PLEG): container finished" podID="addee6de-7eb8-4939-a5f2-966830e970f8" containerID="09b4005651dedb075dfe0f1cc640704f449e9fed97bf82955360a7e2590b04e8" exitCode=0
Nov 02 15:11:29.412145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.411917    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk" event={"ID":"addee6de-7eb8-4939-a5f2-966830e970f8","Type":"ContainerDied","Data":"09b4005651dedb075dfe0f1cc640704f449e9fed97bf82955360a7e2590b04e8"}
Nov 02 15:11:29.499470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.496557    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9\" (UniqueName: \"kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9\") pod \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") "
Nov 02 15:11:29.499470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.496687    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca\") pod \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") "
Nov 02 15:11:29.499470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.496747    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca\") pod \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\" (UID: \"48ec1ef6-d87b-4a4a-af8a-1b9b26aff134\") "
Nov 02 15:11:29.501478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.501409    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca" pod "48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" (UID: "48ec1ef6-d87b-4a4a-af8a-1b9b26aff134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:29.508409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.505783    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9" pod "48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" (UID: "48ec1ef6-d87b-4a4a-af8a-1b9b26aff134"). InnerVolumeSpecName "kube-api-access-m9gp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:29.539601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.509541    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca" pod "48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" (UID: "48ec1ef6-d87b-4a4a-af8a-1b9b26aff134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:29.701484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.699537    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gp9\" (UniqueName: \"kubernetes.io/projected/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-kube-api-access-m9gp9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:29.701484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.699561    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:29.701484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.699572    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:29.903388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.899349    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.903388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:29.899434    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:30.070462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.070352    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.070726 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.070691    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:30.238394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.233848    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.238394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.233908    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:30.704666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.698592    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"
Nov 02 15:11:30.848387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.847692    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.848387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:30.847753    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:31.137136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.137043    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.138483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.138448    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:31.299223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.290261    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.299223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.290317    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:31.310453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.310430    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:31.487402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.474927    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca\") pod \"addee6de-7eb8-4939-a5f2-966830e970f8\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") "
Nov 02 15:11:31.487402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.474992    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca\") pod \"addee6de-7eb8-4939-a5f2-966830e970f8\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") "
Nov 02 15:11:31.487402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.475045    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w\" (UniqueName: \"kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w\") pod \"addee6de-7eb8-4939-a5f2-966830e970f8\" (UID: \"addee6de-7eb8-4939-a5f2-966830e970f8\") "
Nov 02 15:11:31.487402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.485757    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca" pod "addee6de-7eb8-4939-a5f2-966830e970f8" (UID: "addee6de-7eb8-4939-a5f2-966830e970f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:31.499276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.499144    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w" pod "addee6de-7eb8-4939-a5f2-966830e970f8" (UID: "addee6de-7eb8-4939-a5f2-966830e970f8"). InnerVolumeSpecName "kube-api-access-k5j4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:31.500806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.500766    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca" pod "addee6de-7eb8-4939-a5f2-966830e970f8" (UID: "addee6de-7eb8-4939-a5f2-966830e970f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:31.577623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.577595    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:31.578470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.578430    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/addee6de-7eb8-4939-a5f2-966830e970f8-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:31.578470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.578452    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5j4w\" (UniqueName: \"kubernetes.io/projected/addee6de-7eb8-4939-a5f2-966830e970f8-kube-api-access-k5j4w\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:31.774436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.770021    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk" event={"ID":"addee6de-7eb8-4939-a5f2-966830e970f8","Type":"ContainerDied","Data":"1354319db82669785730cd3185dbdeb747d12470ccf99621ea871fd390638445"}
Nov 02 15:11:31.774436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.770087    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1354319db82669785730cd3185dbdeb747d12470ccf99621ea871fd390638445"
Nov 02 15:11:31.774436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.770163    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"
Nov 02 15:11:31.858428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.856748    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.858428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:31.856810    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:32.054914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.054825    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.054914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.054881    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:32.226410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.226346    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.226586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.226425    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 15:11:32.649128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.649088    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"]
Nov 02 15:11:32.650306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.650279    5041 cpu_manager.go:414] "containerMap: removing container" podUID="addee6de-7eb8-4939-a5f2-966830e970f8" containerName="revoke-keys"
Nov 02 15:11:32.650306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.650308    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addee6de-7eb8-4939-a5f2-966830e970f8" containerName="revoke-keys"
Nov 02 15:11:32.650556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.650343    5041 cpu_manager.go:414] "containerMap: removing container" podUID="48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" containerName="revoke-keys"
Nov 02 15:11:32.650556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.650351    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" containerName="revoke-keys"
Nov 02 15:11:32.650678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.650606    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="addee6de-7eb8-4939-a5f2-966830e970f8" containerName="revoke-keys"
Nov 02 15:11:32.650678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.650629    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" containerName="revoke-keys"
Nov 02 15:11:32.655576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.655550    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.684949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.683298    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:32.685341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.685183    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:32.694278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.694252    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"]
Nov 02 15:11:32.806403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.804662    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9g2\" (UniqueName: \"kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.806403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.804717    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.806403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.805073    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.865441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.856620    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.865441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.856678    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 15:11:32.930411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.924096    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.930411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.925848    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9g2\" (UniqueName: \"kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.930411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.926003    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.939200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.939157    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.942337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.941948    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.950349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.950256    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9g2\" (UniqueName: \"kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2\") pod \"maas-api-revoke-sub-1e006390-5rrdx\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:32.988767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:32.988737    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:33.060306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.059298    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 02 15:11:33.060306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.059748    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 02 15:11:33.110827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.110336    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"]
Nov 02 15:11:33.116652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.116613    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.151979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.149577    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"]
Nov 02 15:11:33.294039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.288013    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29tx\" (UniqueName: \"kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.294039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.288303    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.294039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.288398    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.294039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.293451    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:33.294874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.294849    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:33.295060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.295043    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:11:33.419870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.368194    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"]
Nov 02 15:11:33.419870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.408182    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.419870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.408265    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.419870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.408406    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29tx\" (UniqueName: \"kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.419870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.410036    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.440236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.437027    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.472988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.472906    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29tx\" (UniqueName: \"kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx\") pod \"maas-api-revoke-sub-822e0586-vwnkm\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.502187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.501855    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:33.793194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.793152    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx" event={"ID":"962bfa09-6008-4180-85f8-3676289e6f9e","Type":"ContainerStarted","Data":"4084b72a6d613928f18f00509dc453e4cb55256ecb2069772e3c1c75fe78f0ed"}
Nov 02 15:11:33.853344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.853316    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:33.854776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.854714    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:33.855094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.855058    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:11:33.902342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:33.902312    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"]
Nov 02 15:11:34.058624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.058597    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:34.059106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.059088    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:34.059886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.059862    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:11:34.808870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.808831    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm" event={"ID":"144923bb-260c-4905-9489-45062b69a29f","Type":"ContainerStarted","Data":"cac62fa4d43def8b03d8e726661de7611a4c29d03877a8b4195752f7918b44de"}
Nov 02 15:11:34.809060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.808916    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm" event={"ID":"144923bb-260c-4905-9489-45062b69a29f","Type":"ContainerStarted","Data":"e742bc95c80d0354d37320ebe29b7981548cbc97010213d0d3b198b57a3284ea"}
Nov 02 15:11:34.812438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.812356    5041 generic.go:356] "Generic (PLEG): container finished" podID="962bfa09-6008-4180-85f8-3676289e6f9e" containerID="f4ac1f1cc1abea325ce14aa666f1b08714b33fd6614b2c37a33f4eaf7b24152f" exitCode=0
Nov 02 15:11:34.813504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.813475    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx" event={"ID":"962bfa09-6008-4180-85f8-3676289e6f9e","Type":"ContainerDied","Data":"f4ac1f1cc1abea325ce14aa666f1b08714b33fd6614b2c37a33f4eaf7b24152f"}
Nov 02 15:11:34.846267 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:34.846176    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm" podStartSLOduration=1.846163911 podStartE2EDuration="1.846163911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:34.844995398 +0000 UTC m=+947.103064850" watchObservedRunningTime="2026-11-02 15:11:34.846163911 +0000 UTC m=+947.104233361"
Nov 02 15:11:35.276339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.276291    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"]
Nov 02 15:11:35.293156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.292266    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.317383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.302684    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"]
Nov 02 15:11:35.353565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.353532    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.353711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.353572    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qfj\" (UniqueName: \"kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.353776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.353746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.454581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.454499    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.454736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.454682    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.454736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.454715    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qfj\" (UniqueName: \"kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.455374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.455334    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.457352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.457328    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.479464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.479432    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qfj\" (UniqueName: \"kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj\") pod \"maas-api-revoke-sub-0c0d8a77-4fhv8\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.620186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.620111    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:35.825671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.825639    5041 generic.go:356] "Generic (PLEG): container finished" podID="144923bb-260c-4905-9489-45062b69a29f" containerID="cac62fa4d43def8b03d8e726661de7611a4c29d03877a8b4195752f7918b44de" exitCode=0
Nov 02 15:11:35.825843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.825707    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm" event={"ID":"144923bb-260c-4905-9489-45062b69a29f","Type":"ContainerDied","Data":"cac62fa4d43def8b03d8e726661de7611a4c29d03877a8b4195752f7918b44de"}
Nov 02 15:11:35.947659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.944182    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"]
Nov 02 15:11:35.994490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:35.994462    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:36.062654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.062623    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca\") pod \"962bfa09-6008-4180-85f8-3676289e6f9e\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") "
Nov 02 15:11:36.062807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.062720    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca\") pod \"962bfa09-6008-4180-85f8-3676289e6f9e\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") "
Nov 02 15:11:36.062884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.062830    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2\" (UniqueName: \"kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2\") pod \"962bfa09-6008-4180-85f8-3676289e6f9e\" (UID: \"962bfa09-6008-4180-85f8-3676289e6f9e\") "
Nov 02 15:11:36.063048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.063020    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca" pod "962bfa09-6008-4180-85f8-3676289e6f9e" (UID: "962bfa09-6008-4180-85f8-3676289e6f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:36.063523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.063496    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:36.064860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.064833    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca" pod "962bfa09-6008-4180-85f8-3676289e6f9e" (UID: "962bfa09-6008-4180-85f8-3676289e6f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:36.065546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.065522    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2" pod "962bfa09-6008-4180-85f8-3676289e6f9e" (UID: "962bfa09-6008-4180-85f8-3676289e6f9e"). InnerVolumeSpecName "kube-api-access-zg9g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:36.164997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.164930    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962bfa09-6008-4180-85f8-3676289e6f9e-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:36.164997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.164962    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9g2\" (UniqueName: \"kubernetes.io/projected/962bfa09-6008-4180-85f8-3676289e6f9e-kube-api-access-zg9g2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:36.814192 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:11:36.814160    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:11:36.836849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.836830    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"
Nov 02 15:11:36.836935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.836864    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx" event={"ID":"962bfa09-6008-4180-85f8-3676289e6f9e","Type":"ContainerDied","Data":"4084b72a6d613928f18f00509dc453e4cb55256ecb2069772e3c1c75fe78f0ed"}
Nov 02 15:11:36.836935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.836916    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4084b72a6d613928f18f00509dc453e4cb55256ecb2069772e3c1c75fe78f0ed"
Nov 02 15:11:36.838529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.838506    5041 generic.go:356] "Generic (PLEG): container finished" podID="d068689e-1232-43de-b883-baa1fa9f404c" containerID="73a5fc0da003b8f9d534a108863cb93b87360b9bb1ab2714b7d64ceb5aee4e66" exitCode=0
Nov 02 15:11:36.838610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.838584    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8" event={"ID":"d068689e-1232-43de-b883-baa1fa9f404c","Type":"ContainerDied","Data":"73a5fc0da003b8f9d534a108863cb93b87360b9bb1ab2714b7d64ceb5aee4e66"}
Nov 02 15:11:36.838657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:36.838624    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8" event={"ID":"d068689e-1232-43de-b883-baa1fa9f404c","Type":"ContainerStarted","Data":"74a727ab4d9b50fc9b2c5b7efa227badeaf5718c000ad1a746587bfb87e95956"}
Nov 02 15:11:37.022178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.022154    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:37.128473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.094702    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx\" (UniqueName: \"kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx\") pod \"144923bb-260c-4905-9489-45062b69a29f\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") "
Nov 02 15:11:37.148710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.148434    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca\") pod \"144923bb-260c-4905-9489-45062b69a29f\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") "
Nov 02 15:11:37.148710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.148574    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca\") pod \"144923bb-260c-4905-9489-45062b69a29f\" (UID: \"144923bb-260c-4905-9489-45062b69a29f\") "
Nov 02 15:11:37.153378 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.153314    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx" pod "144923bb-260c-4905-9489-45062b69a29f" (UID: "144923bb-260c-4905-9489-45062b69a29f"). InnerVolumeSpecName "kube-api-access-t29tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:37.155500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.155467    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca" pod "144923bb-260c-4905-9489-45062b69a29f" (UID: "144923bb-260c-4905-9489-45062b69a29f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:37.224574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.224536    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca" pod "144923bb-260c-4905-9489-45062b69a29f" (UID: "144923bb-260c-4905-9489-45062b69a29f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:37.251015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.250983    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29tx\" (UniqueName: \"kubernetes.io/projected/144923bb-260c-4905-9489-45062b69a29f-kube-api-access-t29tx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:37.251142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.251019    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144923bb-260c-4905-9489-45062b69a29f-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:37.251142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.251038    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144923bb-260c-4905-9489-45062b69a29f-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:37.848556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.848472    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"
Nov 02 15:11:37.848556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.848492    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm" event={"ID":"144923bb-260c-4905-9489-45062b69a29f","Type":"ContainerDied","Data":"e742bc95c80d0354d37320ebe29b7981548cbc97010213d0d3b198b57a3284ea"}
Nov 02 15:11:37.848556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:37.848550    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e742bc95c80d0354d37320ebe29b7981548cbc97010213d0d3b198b57a3284ea"
Nov 02 15:11:38.031309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.031278    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:38.160764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.160690    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca\") pod \"d068689e-1232-43de-b883-baa1fa9f404c\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") "
Nov 02 15:11:38.160910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.160842    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca\") pod \"d068689e-1232-43de-b883-baa1fa9f404c\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") "
Nov 02 15:11:38.161038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.161019    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj\" (UniqueName: \"kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj\") pod \"d068689e-1232-43de-b883-baa1fa9f404c\" (UID: \"d068689e-1232-43de-b883-baa1fa9f404c\") "
Nov 02 15:11:38.161105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.161084    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca" pod "d068689e-1232-43de-b883-baa1fa9f404c" (UID: "d068689e-1232-43de-b883-baa1fa9f404c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:38.161579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.161553    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:38.162971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.162948    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca" pod "d068689e-1232-43de-b883-baa1fa9f404c" (UID: "d068689e-1232-43de-b883-baa1fa9f404c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:38.163208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.163186    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj" pod "d068689e-1232-43de-b883-baa1fa9f404c" (UID: "d068689e-1232-43de-b883-baa1fa9f404c"). InnerVolumeSpecName "kube-api-access-48qfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:38.262327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.262301    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d068689e-1232-43de-b883-baa1fa9f404c-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:38.262327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.262328    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48qfj\" (UniqueName: \"kubernetes.io/projected/d068689e-1232-43de-b883-baa1fa9f404c-kube-api-access-48qfj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:38.760822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.760788    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"]
Nov 02 15:11:38.761549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761522    5041 cpu_manager.go:414] "containerMap: removing container" podUID="962bfa09-6008-4180-85f8-3676289e6f9e" containerName="revoke-keys"
Nov 02 15:11:38.761549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761550    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962bfa09-6008-4180-85f8-3676289e6f9e" containerName="revoke-keys"
Nov 02 15:11:38.761651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761561    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d068689e-1232-43de-b883-baa1fa9f404c" containerName="revoke-keys"
Nov 02 15:11:38.761651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761567    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d068689e-1232-43de-b883-baa1fa9f404c" containerName="revoke-keys"
Nov 02 15:11:38.761651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761581    5041 cpu_manager.go:414] "containerMap: removing container" podUID="144923bb-260c-4905-9489-45062b69a29f" containerName="revoke-keys"
Nov 02 15:11:38.761651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761586    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144923bb-260c-4905-9489-45062b69a29f" containerName="revoke-keys"
Nov 02 15:11:38.761767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761722    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="962bfa09-6008-4180-85f8-3676289e6f9e" containerName="revoke-keys"
Nov 02 15:11:38.761767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761729    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d068689e-1232-43de-b883-baa1fa9f404c" containerName="revoke-keys"
Nov 02 15:11:38.761767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.761736    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="144923bb-260c-4905-9489-45062b69a29f" containerName="revoke-keys"
Nov 02 15:11:38.765730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.765711    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.788661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.788634    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"]
Nov 02 15:11:38.859177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.859154    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"
Nov 02 15:11:38.859562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.859193    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8" event={"ID":"d068689e-1232-43de-b883-baa1fa9f404c","Type":"ContainerDied","Data":"74a727ab4d9b50fc9b2c5b7efa227badeaf5718c000ad1a746587bfb87e95956"}
Nov 02 15:11:38.859562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.859239    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a727ab4d9b50fc9b2c5b7efa227badeaf5718c000ad1a746587bfb87e95956"
Nov 02 15:11:38.867743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.867716    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.867858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.867840    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.868021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.867995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbpl\" (UniqueName: \"kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.978385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.977393    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.978385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.977529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.978385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.977695    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbpl\" (UniqueName: \"kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:38.987301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:38.979453    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:39.031461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.019904    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:39.031461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.029155    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbpl\" (UniqueName: \"kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl\") pod \"maas-api-revoke-sub-5c0e0184-mtwt4\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:39.081024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.080991    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:39.391013 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:39.390982    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fc208a_fb36_4420_be1a_dbd040b79a99.slice/crio-d8fe5633011567ff4a9e6f55a40cfa1316bb5fc416ade2ac35f0bd58fb3d797b WatchSource:0}: Error finding container d8fe5633011567ff4a9e6f55a40cfa1316bb5fc416ade2ac35f0bd58fb3d797b: Status 404 returned error can't find the container with id d8fe5633011567ff4a9e6f55a40cfa1316bb5fc416ade2ac35f0bd58fb3d797b
Nov 02 15:11:39.394112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.394089    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"]
Nov 02 15:11:39.870356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.870316    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4" event={"ID":"a9fc208a-fb36-4420-be1a-dbd040b79a99","Type":"ContainerStarted","Data":"2ed9d2e3d043a1f281e35751e47f5331b7c1f1a78648ec1d7d090155d650cf4c"}
Nov 02 15:11:39.870730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.870392    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4" event={"ID":"a9fc208a-fb36-4420-be1a-dbd040b79a99","Type":"ContainerStarted","Data":"d8fe5633011567ff4a9e6f55a40cfa1316bb5fc416ade2ac35f0bd58fb3d797b"}
Nov 02 15:11:39.915734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:39.915680    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4" podStartSLOduration=1.915664056 podStartE2EDuration="1.915664056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:39.905517427 +0000 UTC m=+952.163586877" watchObservedRunningTime="2026-11-02 15:11:39.915664056 +0000 UTC m=+952.173733509"
Nov 02 15:11:40.985407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:40.974641    5041 generic.go:356] "Generic (PLEG): container finished" podID="a9fc208a-fb36-4420-be1a-dbd040b79a99" containerID="2ed9d2e3d043a1f281e35751e47f5331b7c1f1a78648ec1d7d090155d650cf4c" exitCode=0
Nov 02 15:11:40.985407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:40.974697    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4" event={"ID":"a9fc208a-fb36-4420-be1a-dbd040b79a99","Type":"ContainerDied","Data":"2ed9d2e3d043a1f281e35751e47f5331b7c1f1a78648ec1d7d090155d650cf4c"}
Nov 02 15:11:42.099386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.097455    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded"
Nov 02 15:11:42.416432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.415481    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": context deadline exceeded"
Nov 02 15:11:42.421289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.417932    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.486548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.486493    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.852759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.852715    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:11:42.861908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.854029    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:11:42.908662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.908619    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.953615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.953552    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.954584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:42.954554    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.141478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.141398    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:43.221680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.221646    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca\") pod \"a9fc208a-fb36-4420-be1a-dbd040b79a99\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") "
Nov 02 15:11:43.222227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.222190    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl\" (UniqueName: \"kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl\") pod \"a9fc208a-fb36-4420-be1a-dbd040b79a99\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") "
Nov 02 15:11:43.223178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.222667    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca\") pod \"a9fc208a-fb36-4420-be1a-dbd040b79a99\" (UID: \"a9fc208a-fb36-4420-be1a-dbd040b79a99\") "
Nov 02 15:11:43.223481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.223063    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca" pod "a9fc208a-fb36-4420-be1a-dbd040b79a99" (UID: "a9fc208a-fb36-4420-be1a-dbd040b79a99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:43.304527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.304479    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.330387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.315579    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.419770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.331916    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:43.560390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.422215    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca" pod "a9fc208a-fb36-4420-be1a-dbd040b79a99" (UID: "a9fc208a-fb36-4420-be1a-dbd040b79a99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:43.560390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.429551    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl" pod "a9fc208a-fb36-4420-be1a-dbd040b79a99" (UID: "a9fc208a-fb36-4420-be1a-dbd040b79a99"). InnerVolumeSpecName "kube-api-access-dhbpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:43.560390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.432473    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.560390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.558531    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:11:43.679442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.666777    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbpl\" (UniqueName: \"kubernetes.io/projected/a9fc208a-fb36-4420-be1a-dbd040b79a99-kube-api-access-dhbpl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:43.679442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.666816    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fc208a-fb36-4420-be1a-dbd040b79a99-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:43.869496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.869446    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.892389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.870779    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:43.916722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.916577    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.963602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.928475    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.963602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.955477    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.993574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:43.993533    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.350500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.343777    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.443389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.418865    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.552660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.552604    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.623342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.623242    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.774390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.751053    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:11:45.038388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.023653    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:11:45.038388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.023718    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:11:45.038388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.023787    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:11:45.038388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.023818    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"]
Nov 02 15:11:45.038388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.962440    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.038388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.881520    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.072391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:44.976441    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.198389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.050177    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"
Nov 02 15:11:45.198389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.006998    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.198389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.057569    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a9fc208a-fb36-4420-be1a-dbd040b79a99" containerName="revoke-keys"
Nov 02 15:11:45.198389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.147131    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fc208a-fb36-4420-be1a-dbd040b79a99" containerName="revoke-keys"
Nov 02 15:11:45.198389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.147442    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fc208a-fb36-4420-be1a-dbd040b79a99" containerName="revoke-keys"
Nov 02 15:11:45.312483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.288501    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:11:45.312483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.288539    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4" event={"ID":"a9fc208a-fb36-4420-be1a-dbd040b79a99","Type":"ContainerDied","Data":"d8fe5633011567ff4a9e6f55a40cfa1316bb5fc416ade2ac35f0bd58fb3d797b"}
Nov 02 15:11:45.312483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.288564    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8fe5633011567ff4a9e6f55a40cfa1316bb5fc416ade2ac35f0bd58fb3d797b"
Nov 02 15:11:45.312483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.289477    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:45.361455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.315190    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:45.552530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.552477    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.560389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.415662    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.589478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.561104    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:11:45.627413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.321494    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:45.779768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:45.779721    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.046478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.046423    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.078391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.047424    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.078700    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.078914    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.079023    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ccs\" (UniqueName: \"kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.112228    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.117447    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.127517    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:46.195400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.127586    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.298444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.297105    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.319395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.313734    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.319395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.313899    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.319395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.313983    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ccs\" (UniqueName: \"kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.319395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.318780    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.494112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.487295    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ccs\" (UniqueName: \"kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.578445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.560908    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.587943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.587888    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.740389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.739842    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"]
Nov 02 15:11:46.781393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.748425    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:11:46.781393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.759160    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46338096-vhq9w\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:46.877763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:46.877671    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:47.121200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.102970    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:11:47.121200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.118074    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.121200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.118409    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.121200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.119881    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.313400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.310460    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.329684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.324873    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:11:47.344691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.344525    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:11:47.345574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.345558    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:11:47.350164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.350136    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:11:47.366848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.366818    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:11:47.371469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.371394    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:11:47.383154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.383093    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"]
Nov 02 15:11:47.390596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.390574    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.394330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.394275    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"]
Nov 02 15:11:47.470858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.470770    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94bq\" (UniqueName: \"kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.471018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.470876    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.471018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.470903    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.573584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.573544    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94bq\" (UniqueName: \"kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.574064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.573616    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.574064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.573649    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.574548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.574521    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.576795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.576759    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.593496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.593475    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"]
Nov 02 15:11:47.593604 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:47.593583    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7953cc_9bbf_427e_a624_e53d759ba346.slice/crio-b995c21f105d78ec9633550accad0e3e206fabb4ae99eb61e6ccd31fadc003a1 WatchSource:0}: Error finding container b995c21f105d78ec9633550accad0e3e206fabb4ae99eb61e6ccd31fadc003a1: Status 404 returned error can't find the container with id b995c21f105d78ec9633550accad0e3e206fabb4ae99eb61e6ccd31fadc003a1
Nov 02 15:11:47.595242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.595222    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94bq\" (UniqueName: \"kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq\") pod \"maas-api-revoke-sub-670b724c-8jzdb\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:47.746441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:47.745718    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:48.121177 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:48.121135    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e48a660_18a0_42a5_aebd_bc8826719e61.slice/crio-29487a338db05e71e3f933360ec7bc1ec89f239a0d54c7ef10b22d78c1c78231 WatchSource:0}: Error finding container 29487a338db05e71e3f933360ec7bc1ec89f239a0d54c7ef10b22d78c1c78231: Status 404 returned error can't find the container with id 29487a338db05e71e3f933360ec7bc1ec89f239a0d54c7ef10b22d78c1c78231
Nov 02 15:11:48.123646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:48.123612    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"]
Nov 02 15:11:48.420155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:48.420119    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb" event={"ID":"3e48a660-18a0-42a5-aebd-bc8826719e61","Type":"ContainerStarted","Data":"29487a338db05e71e3f933360ec7bc1ec89f239a0d54c7ef10b22d78c1c78231"}
Nov 02 15:11:48.421798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:48.421765    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w" event={"ID":"6a7953cc-9bbf-427e-a624-e53d759ba346","Type":"ContainerStarted","Data":"fe620a5e9dbb0f9f71d32e235c45438c5a0907e212ac15fb78b9103115c5b79b"}
Nov 02 15:11:48.421935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:48.421825    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w" event={"ID":"6a7953cc-9bbf-427e-a624-e53d759ba346","Type":"ContainerStarted","Data":"b995c21f105d78ec9633550accad0e3e206fabb4ae99eb61e6ccd31fadc003a1"}
Nov 02 15:11:48.461808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:48.461753    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w" podStartSLOduration=4.461739815 podStartE2EDuration="4.461739815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:48.456938175 +0000 UTC m=+960.715007659" watchObservedRunningTime="2026-11-02 15:11:48.461739815 +0000 UTC m=+960.719809264"
Nov 02 15:11:49.330756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.330724    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"]
Nov 02 15:11:49.335598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.335574    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.360298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.360256    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"]
Nov 02 15:11:49.429105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.429076    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.429271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.429134    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5b8\" (UniqueName: \"kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.429345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.429316    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.433416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.433357    5041 generic.go:356] "Generic (PLEG): container finished" podID="3e48a660-18a0-42a5-aebd-bc8826719e61" containerID="ee0277b70ad257c8f088448586eb1940901bdf24831e090dd10c89a4f5371043" exitCode=0
Nov 02 15:11:49.433548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.433465    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb" event={"ID":"3e48a660-18a0-42a5-aebd-bc8826719e61","Type":"ContainerDied","Data":"ee0277b70ad257c8f088448586eb1940901bdf24831e090dd10c89a4f5371043"}
Nov 02 15:11:49.435120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.435097    5041 generic.go:356] "Generic (PLEG): container finished" podID="6a7953cc-9bbf-427e-a624-e53d759ba346" containerID="fe620a5e9dbb0f9f71d32e235c45438c5a0907e212ac15fb78b9103115c5b79b" exitCode=0
Nov 02 15:11:49.435218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.435174    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w" event={"ID":"6a7953cc-9bbf-427e-a624-e53d759ba346","Type":"ContainerDied","Data":"fe620a5e9dbb0f9f71d32e235c45438c5a0907e212ac15fb78b9103115c5b79b"}
Nov 02 15:11:49.531046 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.531011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.531206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.531175    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.531253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.531216    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5b8\" (UniqueName: \"kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.531690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.531669    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.533584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.533562    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.560924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.560896    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5b8\" (UniqueName: \"kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8\") pod \"maas-api-revoke-sub-4a470704-tn2hc\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.650838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.650794    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:49.979906 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:49.979879    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40abc75_f698_4962_bc7b_539764eddf92.slice/crio-a0982592bc5a52cbc54872e89d2e513eca1d5fdd10d19b2f78a419ab6ce94593 WatchSource:0}: Error finding container a0982592bc5a52cbc54872e89d2e513eca1d5fdd10d19b2f78a419ab6ce94593: Status 404 returned error can't find the container with id a0982592bc5a52cbc54872e89d2e513eca1d5fdd10d19b2f78a419ab6ce94593
Nov 02 15:11:49.982575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:49.982548    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"]
Nov 02 15:11:50.445881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.445843    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc" event={"ID":"c40abc75-f698-4962-bc7b-539764eddf92","Type":"ContainerStarted","Data":"b8bdef89d623d2a5186660ffd3b34230ef629af7ac816024005eaa50d71a6912"}
Nov 02 15:11:50.446307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.445897    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc" event={"ID":"c40abc75-f698-4962-bc7b-539764eddf92","Type":"ContainerStarted","Data":"a0982592bc5a52cbc54872e89d2e513eca1d5fdd10d19b2f78a419ab6ce94593"}
Nov 02 15:11:50.489608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.489316    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc" podStartSLOduration=1.489298791 podStartE2EDuration="1.489298791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:50.48260258 +0000 UTC m=+962.740672030" watchObservedRunningTime="2026-11-02 15:11:50.489298791 +0000 UTC m=+962.747368243"
Nov 02 15:11:50.688823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.688792    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:50.692086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.692065    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:50.809568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.809486    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"]
Nov 02 15:11:50.810229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.810216    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6a7953cc-9bbf-427e-a624-e53d759ba346" containerName="revoke-keys"
Nov 02 15:11:50.810289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.810230    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7953cc-9bbf-427e-a624-e53d759ba346" containerName="revoke-keys"
Nov 02 15:11:50.810289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.810245    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3e48a660-18a0-42a5-aebd-bc8826719e61" containerName="revoke-keys"
Nov 02 15:11:50.810289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.810250    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e48a660-18a0-42a5-aebd-bc8826719e61" containerName="revoke-keys"
Nov 02 15:11:50.810459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.810395    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e48a660-18a0-42a5-aebd-bc8826719e61" containerName="revoke-keys"
Nov 02 15:11:50.810459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.810415    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7953cc-9bbf-427e-a624-e53d759ba346" containerName="revoke-keys"
Nov 02 15:11:50.814291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.814266    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:50.851963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.851931    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca\") pod \"6a7953cc-9bbf-427e-a624-e53d759ba346\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") "
Nov 02 15:11:50.852108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852018    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs\" (UniqueName: \"kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs\") pod \"6a7953cc-9bbf-427e-a624-e53d759ba346\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") "
Nov 02 15:11:50.852108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852069    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca\") pod \"6a7953cc-9bbf-427e-a624-e53d759ba346\" (UID: \"6a7953cc-9bbf-427e-a624-e53d759ba346\") "
Nov 02 15:11:50.852230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852134    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq\" (UniqueName: \"kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq\") pod \"3e48a660-18a0-42a5-aebd-bc8826719e61\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") "
Nov 02 15:11:50.852230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852172    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca\") pod \"3e48a660-18a0-42a5-aebd-bc8826719e61\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") "
Nov 02 15:11:50.852230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852196    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca\") pod \"3e48a660-18a0-42a5-aebd-bc8826719e61\" (UID: \"3e48a660-18a0-42a5-aebd-bc8826719e61\") "
Nov 02 15:11:50.852945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852890    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca" pod "3e48a660-18a0-42a5-aebd-bc8826719e61" (UID: "3e48a660-18a0-42a5-aebd-bc8826719e61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:50.853057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.852988    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca" pod "6a7953cc-9bbf-427e-a624-e53d759ba346" (UID: "6a7953cc-9bbf-427e-a624-e53d759ba346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:50.854861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.854838    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca" pod "6a7953cc-9bbf-427e-a624-e53d759ba346" (UID: "6a7953cc-9bbf-427e-a624-e53d759ba346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:50.855237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.855209    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca" pod "3e48a660-18a0-42a5-aebd-bc8826719e61" (UID: "3e48a660-18a0-42a5-aebd-bc8826719e61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:50.855336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.855235    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq" pod "3e48a660-18a0-42a5-aebd-bc8826719e61" (UID: "3e48a660-18a0-42a5-aebd-bc8826719e61"). InnerVolumeSpecName "kube-api-access-f94bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:50.856497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.856471    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs" pod "6a7953cc-9bbf-427e-a624-e53d759ba346" (UID: "6a7953cc-9bbf-427e-a624-e53d759ba346"). InnerVolumeSpecName "kube-api-access-k9ccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:50.859863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.859839    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"]
Nov 02 15:11:50.954577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.954551    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:50.954731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.954588    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4ph\" (UniqueName: \"kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:50.954811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.954784    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:50.955022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.955004    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:50.955068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.955026    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ccs\" (UniqueName: \"kubernetes.io/projected/6a7953cc-9bbf-427e-a624-e53d759ba346-kube-api-access-k9ccs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:50.955068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.955037    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7953cc-9bbf-427e-a624-e53d759ba346-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:50.955068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.955046    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f94bq\" (UniqueName: \"kubernetes.io/projected/3e48a660-18a0-42a5-aebd-bc8826719e61-kube-api-access-f94bq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:50.955068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.955054    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:50.955068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:50.955063    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e48a660-18a0-42a5-aebd-bc8826719e61-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:51.056137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.056106    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.056288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.056147    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4ph\" (UniqueName: \"kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.056288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.056223    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.056861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.056837    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.058330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.058311    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.077043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.076988    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4ph\" (UniqueName: \"kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph\") pod \"maas-api-revoke-sub-fedebb34-rhpww\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.142398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.141868    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:51.456439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.456405    5041 generic.go:356] "Generic (PLEG): container finished" podID="c40abc75-f698-4962-bc7b-539764eddf92" containerID="b8bdef89d623d2a5186660ffd3b34230ef629af7ac816024005eaa50d71a6912" exitCode=0
Nov 02 15:11:51.456835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.456441    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc" event={"ID":"c40abc75-f698-4962-bc7b-539764eddf92","Type":"ContainerDied","Data":"b8bdef89d623d2a5186660ffd3b34230ef629af7ac816024005eaa50d71a6912"}
Nov 02 15:11:51.458217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.458134    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"
Nov 02 15:11:51.458466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.458147    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb" event={"ID":"3e48a660-18a0-42a5-aebd-bc8826719e61","Type":"ContainerDied","Data":"29487a338db05e71e3f933360ec7bc1ec89f239a0d54c7ef10b22d78c1c78231"}
Nov 02 15:11:51.458603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.458587    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29487a338db05e71e3f933360ec7bc1ec89f239a0d54c7ef10b22d78c1c78231"
Nov 02 15:11:51.460028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.460006    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w" event={"ID":"6a7953cc-9bbf-427e-a624-e53d759ba346","Type":"ContainerDied","Data":"b995c21f105d78ec9633550accad0e3e206fabb4ae99eb61e6ccd31fadc003a1"}
Nov 02 15:11:51.460135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.460028    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"
Nov 02 15:11:51.460135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.460042    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b995c21f105d78ec9633550accad0e3e206fabb4ae99eb61e6ccd31fadc003a1"
Nov 02 15:11:51.524510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.524316    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"]
Nov 02 15:11:51.545388 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:51.544746    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb6e912_0ac6_46cf_9af4_53b713fc74b8.slice/crio-cdc8093380acef822db53974f699091885719c2905dccd7410c9e9a7bccc3be0 WatchSource:0}: Error finding container cdc8093380acef822db53974f699091885719c2905dccd7410c9e9a7bccc3be0: Status 404 returned error can't find the container with id cdc8093380acef822db53974f699091885719c2905dccd7410c9e9a7bccc3be0
Nov 02 15:11:51.980043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.979971    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"]
Nov 02 15:11:51.984844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:51.984823    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.010208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.010174    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"]
Nov 02 15:11:52.089394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.089328    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.089560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.089475    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.089674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.089658    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwpk\" (UniqueName: \"kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.191404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.191355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwpk\" (UniqueName: \"kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.191548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.191477    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.191548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.191519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.192196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.192171    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.194139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.194113    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.213033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.213004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwpk\" (UniqueName: \"kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk\") pod \"maas-api-revoke-sub-a2784718-n7zlg\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.300689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.300614    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:52.470459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.470421    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww" event={"ID":"9eb6e912-0ac6-46cf-9af4-53b713fc74b8","Type":"ContainerStarted","Data":"bcec9ef43f6a2076e809d09a8a90743e35bde71e47968534ee1e9d2548e54b6c"}
Nov 02 15:11:52.470903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.470478    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww" event={"ID":"9eb6e912-0ac6-46cf-9af4-53b713fc74b8","Type":"ContainerStarted","Data":"cdc8093380acef822db53974f699091885719c2905dccd7410c9e9a7bccc3be0"}
Nov 02 15:11:52.506500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.506453    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww" podStartSLOduration=2.506439677 podStartE2EDuration="2.506439677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:52.505716686 +0000 UTC m=+964.763786141" watchObservedRunningTime="2026-11-02 15:11:52.506439677 +0000 UTC m=+964.764509126"
Nov 02 15:11:52.614780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.614750    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"]
Nov 02 15:11:52.627464 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:52.627434    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb14608_6d60_4cb1_a855_b02c707b4f54.slice/crio-747d0ce98b9f70c4e445c67eb028391eb85c93b0fdbad4692a43292bd7851911 WatchSource:0}: Error finding container 747d0ce98b9f70c4e445c67eb028391eb85c93b0fdbad4692a43292bd7851911: Status 404 returned error can't find the container with id 747d0ce98b9f70c4e445c67eb028391eb85c93b0fdbad4692a43292bd7851911
Nov 02 15:11:52.643939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.643912    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:52.698849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.698814    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca\") pod \"c40abc75-f698-4962-bc7b-539764eddf92\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") "
Nov 02 15:11:52.699027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.699011    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca\") pod \"c40abc75-f698-4962-bc7b-539764eddf92\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") "
Nov 02 15:11:52.699144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.699128    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8\" (UniqueName: \"kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8\") pod \"c40abc75-f698-4962-bc7b-539764eddf92\" (UID: \"c40abc75-f698-4962-bc7b-539764eddf92\") "
Nov 02 15:11:52.699455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.699422    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca" pod "c40abc75-f698-4962-bc7b-539764eddf92" (UID: "c40abc75-f698-4962-bc7b-539764eddf92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:52.700158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.700130    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:52.701245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.701218    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca" pod "c40abc75-f698-4962-bc7b-539764eddf92" (UID: "c40abc75-f698-4962-bc7b-539764eddf92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:52.701707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.701678    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8" pod "c40abc75-f698-4962-bc7b-539764eddf92" (UID: "c40abc75-f698-4962-bc7b-539764eddf92"). InnerVolumeSpecName "kube-api-access-ht5b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:52.841758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.812595    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40abc75-f698-4962-bc7b-539764eddf92-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:52.841758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:52.812639    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5b8\" (UniqueName: \"kubernetes.io/projected/c40abc75-f698-4962-bc7b-539764eddf92-kube-api-access-ht5b8\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:53.480693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.480656    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc" event={"ID":"c40abc75-f698-4962-bc7b-539764eddf92","Type":"ContainerDied","Data":"a0982592bc5a52cbc54872e89d2e513eca1d5fdd10d19b2f78a419ab6ce94593"}
Nov 02 15:11:53.481116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.480736    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0982592bc5a52cbc54872e89d2e513eca1d5fdd10d19b2f78a419ab6ce94593"
Nov 02 15:11:53.481116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.480669    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"
Nov 02 15:11:53.482378 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.482344    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg" event={"ID":"0eb14608-6d60-4cb1-a855-b02c707b4f54","Type":"ContainerStarted","Data":"d1c3d351e031ac0b30a6a2bcdb79e3aaa7749aa90653c555c8644b113cb2a132"}
Nov 02 15:11:53.482493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.482418    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg" event={"ID":"0eb14608-6d60-4cb1-a855-b02c707b4f54","Type":"ContainerStarted","Data":"747d0ce98b9f70c4e445c67eb028391eb85c93b0fdbad4692a43292bd7851911"}
Nov 02 15:11:53.484103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.484083    5041 generic.go:356] "Generic (PLEG): container finished" podID="9eb6e912-0ac6-46cf-9af4-53b713fc74b8" containerID="bcec9ef43f6a2076e809d09a8a90743e35bde71e47968534ee1e9d2548e54b6c" exitCode=0
Nov 02 15:11:53.484190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.484167    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww" event={"ID":"9eb6e912-0ac6-46cf-9af4-53b713fc74b8","Type":"ContainerDied","Data":"bcec9ef43f6a2076e809d09a8a90743e35bde71e47968534ee1e9d2548e54b6c"}
Nov 02 15:11:53.522029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:53.521984    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg" podStartSLOduration=2.521972517 podStartE2EDuration="2.521972517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:53.51784848 +0000 UTC m=+965.775917931" watchObservedRunningTime="2026-11-02 15:11:53.521972517 +0000 UTC m=+965.780041967"
Nov 02 15:11:54.197013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.196978    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"]
Nov 02 15:11:54.197901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.197881    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c40abc75-f698-4962-bc7b-539764eddf92" containerName="revoke-keys"
Nov 02 15:11:54.197975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.197902    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40abc75-f698-4962-bc7b-539764eddf92" containerName="revoke-keys"
Nov 02 15:11:54.198043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.198033    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40abc75-f698-4962-bc7b-539764eddf92" containerName="revoke-keys"
Nov 02 15:11:54.202097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.202078    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232336    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232375    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232400    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"]
Nov 02 15:11:54.232460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232430    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knfk\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232576    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232733    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232848    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.232998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.232963    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334024    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334095    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334249    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334329    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334429    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334513    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knfk\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334574    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334644    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334673    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334654    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334745    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334909    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.334980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.334971    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.336875 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.336854    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.336984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.336951    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.356615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.356590    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.356761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.356741    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knfk\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.520325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.519941    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:54.540338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.540300    5041 generic.go:356] "Generic (PLEG): container finished" podID="0eb14608-6d60-4cb1-a855-b02c707b4f54" containerID="d1c3d351e031ac0b30a6a2bcdb79e3aaa7749aa90653c555c8644b113cb2a132" exitCode=0
Nov 02 15:11:54.540594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.540555    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg" event={"ID":"0eb14608-6d60-4cb1-a855-b02c707b4f54","Type":"ContainerDied","Data":"d1c3d351e031ac0b30a6a2bcdb79e3aaa7749aa90653c555c8644b113cb2a132"}
Nov 02 15:11:54.795538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.795483    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:54.843738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.843705    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca\") pod \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") "
Nov 02 15:11:54.843938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.843847    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca\") pod \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") "
Nov 02 15:11:54.844081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.844064    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph\" (UniqueName: \"kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph\") pod \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\" (UID: \"9eb6e912-0ac6-46cf-9af4-53b713fc74b8\") "
Nov 02 15:11:54.844519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.844400    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca" pod "9eb6e912-0ac6-46cf-9af4-53b713fc74b8" (UID: "9eb6e912-0ac6-46cf-9af4-53b713fc74b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:54.845073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.845048    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:54.846765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.846728    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca" pod "9eb6e912-0ac6-46cf-9af4-53b713fc74b8" (UID: "9eb6e912-0ac6-46cf-9af4-53b713fc74b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:54.846919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.846869    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph" pod "9eb6e912-0ac6-46cf-9af4-53b713fc74b8" (UID: "9eb6e912-0ac6-46cf-9af4-53b713fc74b8"). InnerVolumeSpecName "kube-api-access-4w4ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:54.881411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.881385    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"]
Nov 02 15:11:54.888431 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:11:54.888403    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1fd59e_e864_4bc6_934e_af5327b9f84f.slice/crio-40acf5d5a07a6fd6e506cb873ed8c311ac324e2b0bb9d28cc727346e9d1c2fdf WatchSource:0}: Error finding container 40acf5d5a07a6fd6e506cb873ed8c311ac324e2b0bb9d28cc727346e9d1c2fdf: Status 404 returned error can't find the container with id 40acf5d5a07a6fd6e506cb873ed8c311ac324e2b0bb9d28cc727346e9d1c2fdf
Nov 02 15:11:54.890094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.890051    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:54.890181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.890139    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:54.890226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.890189    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:54.946506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.946479    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:54.946587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:54.946512    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4ph\" (UniqueName: \"kubernetes.io/projected/9eb6e912-0ac6-46cf-9af4-53b713fc74b8-kube-api-access-4w4ph\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:55.552025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.551989    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"
Nov 02 15:11:55.552477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.551991    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww" event={"ID":"9eb6e912-0ac6-46cf-9af4-53b713fc74b8","Type":"ContainerDied","Data":"cdc8093380acef822db53974f699091885719c2905dccd7410c9e9a7bccc3be0"}
Nov 02 15:11:55.552477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.552161    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc8093380acef822db53974f699091885719c2905dccd7410c9e9a7bccc3be0"
Nov 02 15:11:55.554086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.554058    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" event={"ID":"2b1fd59e-e864-4bc6-934e-af5327b9f84f","Type":"ContainerStarted","Data":"f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404"}
Nov 02 15:11:55.554251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.554105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" event={"ID":"2b1fd59e-e864-4bc6-934e-af5327b9f84f","Type":"ContainerStarted","Data":"40acf5d5a07a6fd6e506cb873ed8c311ac324e2b0bb9d28cc727346e9d1c2fdf"}
Nov 02 15:11:55.600842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.600790    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podStartSLOduration=1.600773475 podStartE2EDuration="1.600773475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:55.596761411 +0000 UTC m=+967.854830873" watchObservedRunningTime="2026-11-02 15:11:55.600773475 +0000 UTC m=+967.858842925"
Nov 02 15:11:55.849449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.849377    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:55.962626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.962552    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca\") pod \"0eb14608-6d60-4cb1-a855-b02c707b4f54\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") "
Nov 02 15:11:55.962850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.962678    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk\" (UniqueName: \"kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk\") pod \"0eb14608-6d60-4cb1-a855-b02c707b4f54\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") "
Nov 02 15:11:55.962850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.962735    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca\") pod \"0eb14608-6d60-4cb1-a855-b02c707b4f54\" (UID: \"0eb14608-6d60-4cb1-a855-b02c707b4f54\") "
Nov 02 15:11:55.963311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.963266    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca" pod "0eb14608-6d60-4cb1-a855-b02c707b4f54" (UID: "0eb14608-6d60-4cb1-a855-b02c707b4f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:55.963484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.963460    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:55.964987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.964961    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca" pod "0eb14608-6d60-4cb1-a855-b02c707b4f54" (UID: "0eb14608-6d60-4cb1-a855-b02c707b4f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:55.965321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:55.965295    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk" pod "0eb14608-6d60-4cb1-a855-b02c707b4f54" (UID: "0eb14608-6d60-4cb1-a855-b02c707b4f54"). InnerVolumeSpecName "kube-api-access-mfwpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:56.064558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.064526    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb14608-6d60-4cb1-a855-b02c707b4f54-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:56.064558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.064555    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwpk\" (UniqueName: \"kubernetes.io/projected/0eb14608-6d60-4cb1-a855-b02c707b4f54-kube-api-access-mfwpk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:11:56.532388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.521653    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:11:56.537594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.537556    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:11:56.537688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.537622    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 15:11:56.566223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.566200    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"
Nov 02 15:11:56.566668 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.566216    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg" event={"ID":"0eb14608-6d60-4cb1-a855-b02c707b4f54","Type":"ContainerDied","Data":"747d0ce98b9f70c4e445c67eb028391eb85c93b0fdbad4692a43292bd7851911"}
Nov 02 15:11:56.566668 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:56.566264    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="747d0ce98b9f70c4e445c67eb028391eb85c93b0fdbad4692a43292bd7851911"
Nov 02 15:11:57.534835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:57.534794    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:11:57.535119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:57.535090    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 15:11:58.537666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:58.537619    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:11:58.538179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:58.537685    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 15:11:59.522391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:59.520801    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:11:59.522391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:11:59.520868    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 15:12:00.448025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.447987    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"]
Nov 02 15:12:00.448719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.448703    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9eb6e912-0ac6-46cf-9af4-53b713fc74b8" containerName="revoke-keys"
Nov 02 15:12:00.448775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.448720    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb6e912-0ac6-46cf-9af4-53b713fc74b8" containerName="revoke-keys"
Nov 02 15:12:00.448775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.448767    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0eb14608-6d60-4cb1-a855-b02c707b4f54" containerName="revoke-keys"
Nov 02 15:12:00.448775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.448773    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb14608-6d60-4cb1-a855-b02c707b4f54" containerName="revoke-keys"
Nov 02 15:12:00.448884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.448873    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb14608-6d60-4cb1-a855-b02c707b4f54" containerName="revoke-keys"
Nov 02 15:12:00.448921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.448886    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb6e912-0ac6-46cf-9af4-53b713fc74b8" containerName="revoke-keys"
Nov 02 15:12:00.454124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.454106    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.468442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.468417    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"]
Nov 02 15:12:00.482336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.482307    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:00.482336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.482319    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:12:00.537917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.537871    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:12:00.538124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.537956    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 15:12:00.570578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.570542    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42f4f\" (UniqueName: \"kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.570716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.570614    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.570871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.570842    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.672142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.672102    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42f4f\" (UniqueName: \"kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.672306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.672183    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.672306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.672299    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.672966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.672939    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.674711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.674688    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.693684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.693658    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42f4f\" (UniqueName: \"kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f\") pod \"maas-api-revoke-sub-ddd553ee-hx82r\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:00.774423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:00.774333    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:01.089856 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:01.089826    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed8d345_257e_4bf2_9ac0_a14c29b4e00a.slice/crio-d775b7974b36ec018216f6024975ce2ee4ecf7d5fd8b89ffa5eafb53390c239c WatchSource:0}: Error finding container d775b7974b36ec018216f6024975ce2ee4ecf7d5fd8b89ffa5eafb53390c239c: Status 404 returned error can't find the container with id d775b7974b36ec018216f6024975ce2ee4ecf7d5fd8b89ffa5eafb53390c239c
Nov 02 15:12:01.092182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:01.092138    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"]
Nov 02 15:12:01.522319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:01.522287    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:12:01.522745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:01.522347    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 15:12:01.677387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:01.674473    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r" event={"ID":"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a","Type":"ContainerStarted","Data":"dea82b4aef903ac71cc7fd74308400ecc44b431198b8e8554f4ed84cab3a3846"}
Nov 02 15:12:01.677387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:01.674565    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r" event={"ID":"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a","Type":"ContainerStarted","Data":"d775b7974b36ec018216f6024975ce2ee4ecf7d5fd8b89ffa5eafb53390c239c"}
Nov 02 15:12:01.729147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:01.726078    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r" podStartSLOduration=1.72605586 podStartE2EDuration="1.72605586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:01.724636835 +0000 UTC m=+973.982706286" watchObservedRunningTime="2026-11-02 15:12:01.72605586 +0000 UTC m=+973.984125307"
Nov 02 15:12:02.525835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:02.525801    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:12:02.526358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:02.526330    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:12:02.527325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:02.527303    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:12:02.692927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:02.692893    5041 generic.go:356] "Generic (PLEG): container finished" podID="6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" containerID="dea82b4aef903ac71cc7fd74308400ecc44b431198b8e8554f4ed84cab3a3846" exitCode=0
Nov 02 15:12:02.693107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:02.693039    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r" event={"ID":"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a","Type":"ContainerDied","Data":"dea82b4aef903ac71cc7fd74308400ecc44b431198b8e8554f4ed84cab3a3846"}
Nov 02 15:12:03.617786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.617740    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6909-29464c"]
Nov 02 15:12:03.621623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.621602    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:03.646529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.646503    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcqh6\""
Nov 02 15:12:03.652234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.652208    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6909-29464c"]
Nov 02 15:12:03.746986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.746952    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55nw\" (UniqueName: \"kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw\") pod \"test-curl-6909-29464c\" (UID: \"b0c9bd54-2819-409f-8c97-4f2b8de9366f\") " pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:03.848124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.848089    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55nw\" (UniqueName: \"kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw\") pod \"test-curl-6909-29464c\" (UID: \"b0c9bd54-2819-409f-8c97-4f2b8de9366f\") " pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:03.870078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.869997    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55nw\" (UniqueName: \"kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw\") pod \"test-curl-6909-29464c\" (UID: \"b0c9bd54-2819-409f-8c97-4f2b8de9366f\") " pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:03.872908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.872884    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:03.933746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.933707    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:03.952507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.952471    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca\") pod \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") "
Nov 02 15:12:03.952638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.952534    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f\" (UniqueName: \"kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f\") pod \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") "
Nov 02 15:12:03.952638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.952575    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca\") pod \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\" (UID: \"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a\") "
Nov 02 15:12:03.953454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.953417    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca" pod "6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" (UID: "6ed8d345-257e-4bf2-9ac0-a14c29b4e00a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:03.955283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.955254    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f" pod "6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" (UID: "6ed8d345-257e-4bf2-9ac0-a14c29b4e00a"). InnerVolumeSpecName "kube-api-access-42f4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:03.955540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:03.955348    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca" pod "6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" (UID: "6ed8d345-257e-4bf2-9ac0-a14c29b4e00a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:04.054925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.054867    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42f4f\" (UniqueName: \"kubernetes.io/projected/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-kube-api-access-42f4f\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:04.054925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.054904    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:04.054925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.054918    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:04.262156 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:04.262118    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c9bd54_2819_409f_8c97_4f2b8de9366f.slice/crio-1bd32dececf8d54aa677aeaf5c6da63b5181dce369c8c5d31ce4cacd449d5dfa WatchSource:0}: Error finding container 1bd32dececf8d54aa677aeaf5c6da63b5181dce369c8c5d31ce4cacd449d5dfa: Status 404 returned error can't find the container with id 1bd32dececf8d54aa677aeaf5c6da63b5181dce369c8c5d31ce4cacd449d5dfa
Nov 02 15:12:04.264058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.264023    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6909-29464c"]
Nov 02 15:12:04.713229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.713189    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"
Nov 02 15:12:04.713229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.713222    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r" event={"ID":"6ed8d345-257e-4bf2-9ac0-a14c29b4e00a","Type":"ContainerDied","Data":"d775b7974b36ec018216f6024975ce2ee4ecf7d5fd8b89ffa5eafb53390c239c"}
Nov 02 15:12:04.713789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.713270    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d775b7974b36ec018216f6024975ce2ee4ecf7d5fd8b89ffa5eafb53390c239c"
Nov 02 15:12:04.714789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:04.714763    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6909-29464c" event={"ID":"b0c9bd54-2819-409f-8c97-4f2b8de9366f","Type":"ContainerStarted","Data":"1bd32dececf8d54aa677aeaf5c6da63b5181dce369c8c5d31ce4cacd449d5dfa"}
Nov 02 15:12:05.941927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.941877    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"]
Nov 02 15:12:05.943343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.943315    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" containerName="revoke-keys"
Nov 02 15:12:05.943343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.943342    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" containerName="revoke-keys"
Nov 02 15:12:05.943600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.943568    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" containerName="revoke-keys"
Nov 02 15:12:05.947324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.947302    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:05.966342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.966308    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:05.967421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.967338    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:12:05.967560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:05.967531    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"]
Nov 02 15:12:06.031432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.031387    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4ld\" (UniqueName: \"kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.031580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.031455    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.031629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.031613    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.138268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.138222    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.138530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.138509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4ld\" (UniqueName: \"kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.138600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.138557    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.140520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.140490    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.152811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.152780    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.162741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.162708    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4ld\" (UniqueName: \"kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld\") pod \"maas-api-revoke-sub-c4d675bd-6955r\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.265928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.265626    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:06.590417 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:06.590347    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c68130_b369_406f_a46e_0ab6c08d56c8.slice/crio-0f5ef738467df067d03b73913c867d77b598930e2c5ad3725573aa6efc77e84d WatchSource:0}: Error finding container 0f5ef738467df067d03b73913c867d77b598930e2c5ad3725573aa6efc77e84d: Status 404 returned error can't find the container with id 0f5ef738467df067d03b73913c867d77b598930e2c5ad3725573aa6efc77e84d
Nov 02 15:12:06.593272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.593245    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"]
Nov 02 15:12:06.736978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:06.736944    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r" event={"ID":"69c68130-b369-406f-a46e-0ab6c08d56c8","Type":"ContainerStarted","Data":"0f5ef738467df067d03b73913c867d77b598930e2c5ad3725573aa6efc77e84d"}
Nov 02 15:12:06.772002 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:06.771974    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:12:07.747820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:07.747783    5041 generic.go:356] "Generic (PLEG): container finished" podID="69c68130-b369-406f-a46e-0ab6c08d56c8" containerID="22f3e863285a24f912fb9e1c3fbebc81382b7b8dd2738b232627869da9f43acc" exitCode=0
Nov 02 15:12:07.748192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:07.747860    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r" event={"ID":"69c68130-b369-406f-a46e-0ab6c08d56c8","Type":"ContainerDied","Data":"22f3e863285a24f912fb9e1c3fbebc81382b7b8dd2738b232627869da9f43acc"}
Nov 02 15:12:07.749637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:07.749612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6909-29464c" event={"ID":"b0c9bd54-2819-409f-8c97-4f2b8de9366f","Type":"ContainerStarted","Data":"f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493"}
Nov 02 15:12:07.818433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:07.818272    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6909-29464c" podStartSLOduration=2.326961757 podStartE2EDuration="4.818258626s" totalImagesPullingTime="2.491296869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:07.817773312 +0000 UTC m=+980.075842753" watchObservedRunningTime="2026-11-02 15:12:07.818258626 +0000 UTC m=+980.076328075"
Nov 02 15:12:08.970441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:08.970413    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:09.039482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.039396    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"]
Nov 02 15:12:09.040561 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.040435    5041 cpu_manager.go:414] "containerMap: removing container" podUID="69c68130-b369-406f-a46e-0ab6c08d56c8" containerName="revoke-keys"
Nov 02 15:12:09.040561 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.040458    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c68130-b369-406f-a46e-0ab6c08d56c8" containerName="revoke-keys"
Nov 02 15:12:09.041069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.040695    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c68130-b369-406f-a46e-0ab6c08d56c8" containerName="revoke-keys"
Nov 02 15:12:09.044613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.044596    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.067277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.067249    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"]
Nov 02 15:12:09.082155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.082122    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca\") pod \"69c68130-b369-406f-a46e-0ab6c08d56c8\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") "
Nov 02 15:12:09.082489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.082462    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca\") pod \"69c68130-b369-406f-a46e-0ab6c08d56c8\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") "
Nov 02 15:12:09.082959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.082933    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld\" (UniqueName: \"kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld\") pod \"69c68130-b369-406f-a46e-0ab6c08d56c8\" (UID: \"69c68130-b369-406f-a46e-0ab6c08d56c8\") "
Nov 02 15:12:09.083340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.083478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083418    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.083586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083568    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.083695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.083770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083721    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.084444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083818    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.084444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083851    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.084444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083965    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.084444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.083995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqkx\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.084667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.084438    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca" pod "69c68130-b369-406f-a46e-0ab6c08d56c8" (UID: "69c68130-b369-406f-a46e-0ab6c08d56c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:09.091519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.088903    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld" pod "69c68130-b369-406f-a46e-0ab6c08d56c8" (UID: "69c68130-b369-406f-a46e-0ab6c08d56c8"). InnerVolumeSpecName "kube-api-access-wb4ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:09.091519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.090862    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca" pod "69c68130-b369-406f-a46e-0ab6c08d56c8" (UID: "69c68130-b369-406f-a46e-0ab6c08d56c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:09.185586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185548    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185632    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185669    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185752    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185891    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185925    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqkx\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.185980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.185966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.186078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186018    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.186131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186092    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.186191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186173    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:09.186231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186189    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c68130-b369-406f-a46e-0ab6c08d56c8-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:09.186231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186204    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4ld\" (UniqueName: \"kubernetes.io/projected/69c68130-b369-406f-a46e-0ab6c08d56c8-kube-api-access-wb4ld\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:09.186448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.186544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186456    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.186544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186506    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.186743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.186721    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.188204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.188177    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.188301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.188178    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.211807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.211779    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqkx\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.212566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.212546    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.359757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.359667    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:09.686718 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:09.686687    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba23440_bf0b_40e7_9314_63c85090cacd.slice/crio-e932f940aeb7ba60ecde3c80c0834ca858a795df4df678052917d4d04218dd9d WatchSource:0}: Error finding container e932f940aeb7ba60ecde3c80c0834ca858a795df4df678052917d4d04218dd9d: Status 404 returned error can't find the container with id e932f940aeb7ba60ecde3c80c0834ca858a795df4df678052917d4d04218dd9d
Nov 02 15:12:09.691327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.691291    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"]
Nov 02 15:12:09.691866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.691819    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:09.691986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.691966    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:09.692078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.692056    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:09.787900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.787855    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"
Nov 02 15:12:09.787900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.787877    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r" event={"ID":"69c68130-b369-406f-a46e-0ab6c08d56c8","Type":"ContainerDied","Data":"0f5ef738467df067d03b73913c867d77b598930e2c5ad3725573aa6efc77e84d"}
Nov 02 15:12:09.788227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.787944    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5ef738467df067d03b73913c867d77b598930e2c5ad3725573aa6efc77e84d"
Nov 02 15:12:09.791255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.791220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" event={"ID":"8ba23440-bf0b-40e7-9314-63c85090cacd","Type":"ContainerStarted","Data":"153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0"}
Nov 02 15:12:09.791411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.791278    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" event={"ID":"8ba23440-bf0b-40e7-9314-63c85090cacd","Type":"ContainerStarted","Data":"e932f940aeb7ba60ecde3c80c0834ca858a795df4df678052917d4d04218dd9d"}
Nov 02 15:12:09.846412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.845292    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podStartSLOduration=0.845273432 podStartE2EDuration="845.273432ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:09.838695314 +0000 UTC m=+982.096764765" watchObservedRunningTime="2026-11-02 15:12:09.845273432 +0000 UTC m=+982.103342883"
Nov 02 15:12:09.892338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.892303    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"]
Nov 02 15:12:09.897322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.897290    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:09.927690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:09.926886    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"]
Nov 02 15:12:10.001108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001010    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001168    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001273    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001347    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001425    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001555    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrnzm\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001660    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.001826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001795    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.002247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.001975    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.038252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.038209    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"]
Nov 02 15:12:10.043668 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.043634    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.056670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.056640    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:10.056670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.056651    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:12:10.079440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.069154    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"]
Nov 02 15:12:10.111457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111413    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.111728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111530    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.111728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111575    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.111728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111621    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbgf\" (UniqueName: \"kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.111728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111668    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.111728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111709    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.112092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111753    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.112092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111825    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrnzm\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.112092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.111914    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.112092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.112004    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.112092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.112071    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.112531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.112103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.113031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.113004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.113317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.113296    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.114422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.114058    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.114422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.114325    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.114592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.114457    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.118209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.117949    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.127910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.120453    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.138216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.138183    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.142089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.142013    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrnzm\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.213546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.213510    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbgf\" (UniqueName: \"kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.213736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.213702    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.213806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.213757    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.214672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.214639    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.216393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.216352    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.234077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.234044    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:10.235004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.234979    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbgf\" (UniqueName: \"kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf\") pod \"maas-api-revoke-sub-1aba89bf-8sgp2\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.360421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.360343    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:10.361778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.361744    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:10.361912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.361809    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:10.373482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.373455    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:10.721195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.715820    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"]
Nov 02 15:12:10.732345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.731777    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:10.732345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.731873    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:10.732345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.731919    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:10.734521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.734489    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"]
Nov 02 15:12:10.804882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.804851    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" event={"ID":"6a3c9823-4ec5-443a-86a2-dba33a1921cd","Type":"ContainerStarted","Data":"ec0884d01d14778179011f08ca62b7478097b8b36717f6f042ab693b81e26320"}
Nov 02 15:12:10.806587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.806549    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2" event={"ID":"91ea7178-3cbb-4f92-bb30-597ef2c0332c","Type":"ContainerStarted","Data":"005711c65c6c0f332e697c2ad9dabbad83df772053ffe428375f3bd2e4e56a27"}
Nov 02 15:12:10.934045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.932669    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"]
Nov 02 15:12:10.951042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.951011    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"]
Nov 02 15:12:10.951214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:10.951124    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.049472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.049424    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.049900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.049647    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhtf\" (UniqueName: \"kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.049900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.049773    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.153396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.150767    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.153396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.150878    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.153396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.150990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhtf\" (UniqueName: \"kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.153396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.151849    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.220423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.218755    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhtf\" (UniqueName: \"kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.225447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.221756    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0413335b-jcr5c\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.281017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.280987    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:11.360947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.360910    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:11.361111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.360964    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:11.546964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.546877    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"]
Nov 02 15:12:11.552857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.552832    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.572197 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.572174    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"]
Nov 02 15:12:11.659056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.658689    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"]
Nov 02 15:12:11.659663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.659622    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.659774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.659713    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.659962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.659943    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2pm\" (UniqueName: \"kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.770413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.770329    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2pm\" (UniqueName: \"kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.770632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.770488    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.770632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.770531    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.772383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.772105    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.829386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.829289    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.862634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.832991    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2pm\" (UniqueName: \"kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm\") pod \"maas-api-revoke-sub-85fddf9a-9wtf2\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.879251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.879208    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" event={"ID":"6a3c9823-4ec5-443a-86a2-dba33a1921cd","Type":"ContainerStarted","Data":"bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad"}
Nov 02 15:12:11.880049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.879886    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:11.891982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.891957    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2" event={"ID":"91ea7178-3cbb-4f92-bb30-597ef2c0332c","Type":"ContainerStarted","Data":"ee4ae6f760302dc0fc9857b8ed8b7ab2cb1e01069d3cbd391de83f136ba90670"}
Nov 02 15:12:11.892078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.891996    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c" event={"ID":"84d2e9f6-a59f-4c41-ad1f-c282c1c669de","Type":"ContainerStarted","Data":"1831c78c0f50d4ee997edbd7b0f8ea48c3bcc38f3ddfcd72093815899936d878"}
Nov 02 15:12:11.980848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:11.977666    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podStartSLOduration=2.977645446 podStartE2EDuration="2.977645446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:11.952966948 +0000 UTC m=+984.211036395" watchObservedRunningTime="2026-11-02 15:12:11.977645446 +0000 UTC m=+984.235714899"
Nov 02 15:12:12.006057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.003661    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2" podStartSLOduration=2.003642218 podStartE2EDuration="2.003642218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:12.001838437 +0000 UTC m=+984.259907888" watchObservedRunningTime="2026-11-02 15:12:12.003642218 +0000 UTC m=+984.261711649"
Nov 02 15:12:12.280401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.274023    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:12.280401 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:12.274290    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fcfacc_200c_4487_b4dd_6e6d6eedfcb4.slice/crio-f288d227c3a7defd18d3e5ab0cb8a1cd4794e72bd09f6aa2ae3572167ca44ed5 WatchSource:0}: Error finding container f288d227c3a7defd18d3e5ab0cb8a1cd4794e72bd09f6aa2ae3572167ca44ed5: Status 404 returned error can't find the container with id f288d227c3a7defd18d3e5ab0cb8a1cd4794e72bd09f6aa2ae3572167ca44ed5
Nov 02 15:12:12.280401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.275976    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:12.280401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.276011    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:12.285938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.285909    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"]
Nov 02 15:12:12.371842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.371778    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:12.371842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.371836    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:12.893497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.893470    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2" event={"ID":"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4","Type":"ContainerStarted","Data":"82c92272bbd6bd2ba9f5a10d0eb389ecc36425b2ae7aba54aaaeb94dc6e9e0e3"}
Nov 02 15:12:12.893624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.893519    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2" event={"ID":"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4","Type":"ContainerStarted","Data":"f288d227c3a7defd18d3e5ab0cb8a1cd4794e72bd09f6aa2ae3572167ca44ed5"}
Nov 02 15:12:12.895183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.895153    5041 generic.go:356] "Generic (PLEG): container finished" podID="84d2e9f6-a59f-4c41-ad1f-c282c1c669de" containerID="bd25c6fadc197d77930b93d97e2cfd2525c8fef7030736db2693850a4c6e2d62" exitCode=0
Nov 02 15:12:12.895316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.895292    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c" event={"ID":"84d2e9f6-a59f-4c41-ad1f-c282c1c669de","Type":"ContainerDied","Data":"bd25c6fadc197d77930b93d97e2cfd2525c8fef7030736db2693850a4c6e2d62"}
Nov 02 15:12:12.897437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.897420    5041 generic.go:356] "Generic (PLEG): container finished" podID="91ea7178-3cbb-4f92-bb30-597ef2c0332c" containerID="ee4ae6f760302dc0fc9857b8ed8b7ab2cb1e01069d3cbd391de83f136ba90670" exitCode=0
Nov 02 15:12:12.897595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:12.897508    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2" event={"ID":"91ea7178-3cbb-4f92-bb30-597ef2c0332c","Type":"ContainerDied","Data":"ee4ae6f760302dc0fc9857b8ed8b7ab2cb1e01069d3cbd391de83f136ba90670"}
Nov 02 15:12:13.002186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.002130    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2" podStartSLOduration=2.002114441 podStartE2EDuration="2.002114441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:12.961652579 +0000 UTC m=+985.219722031" watchObservedRunningTime="2026-11-02 15:12:13.002114441 +0000 UTC m=+985.260183910"
Nov 02 15:12:13.246386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.245989    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:13.246386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.246049    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:13.363430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.360771    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:13.363430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.360838    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:13.908398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.908293    5041 generic.go:356] "Generic (PLEG): container finished" podID="48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" containerID="82c92272bbd6bd2ba9f5a10d0eb389ecc36425b2ae7aba54aaaeb94dc6e9e0e3" exitCode=0
Nov 02 15:12:13.908560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:13.908390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2" event={"ID":"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4","Type":"ContainerDied","Data":"82c92272bbd6bd2ba9f5a10d0eb389ecc36425b2ae7aba54aaaeb94dc6e9e0e3"}
Nov 02 15:12:14.243386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:14.240209    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:14.243386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:14.240345    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:14.394448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:14.380162    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:14.394448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:14.380247    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:15.213389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.200800    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:15.231235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.231188    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:15.268389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.266524    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:15.268389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.266595    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:15.326511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.326462    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2" event={"ID":"91ea7178-3cbb-4f92-bb30-597ef2c0332c","Type":"ContainerDied","Data":"005711c65c6c0f332e697c2ad9dabbad83df772053ffe428375f3bd2e4e56a27"}
Nov 02 15:12:15.338705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.338671    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005711c65c6c0f332e697c2ad9dabbad83df772053ffe428375f3bd2e4e56a27"
Nov 02 15:12:15.338894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.330546    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"
Nov 02 15:12:15.342276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.342240    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c" event={"ID":"84d2e9f6-a59f-4c41-ad1f-c282c1c669de","Type":"ContainerDied","Data":"1831c78c0f50d4ee997edbd7b0f8ea48c3bcc38f3ddfcd72093815899936d878"}
Nov 02 15:12:15.342507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.342486    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1831c78c0f50d4ee997edbd7b0f8ea48c3bcc38f3ddfcd72093815899936d878"
Nov 02 15:12:15.377614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.377578    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:15.378682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.378649    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:15.379498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.378909    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:15.382055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.382027    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:15.383323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.383296    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:15.403064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.403023    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca\") pod \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") "
Nov 02 15:12:15.403804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.403776    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca\") pod \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") "
Nov 02 15:12:15.403981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.403962    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf\" (UniqueName: \"kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf\") pod \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\" (UID: \"91ea7178-3cbb-4f92-bb30-597ef2c0332c\") "
Nov 02 15:12:15.404114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.404095    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca\") pod \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") "
Nov 02 15:12:15.404439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.404416    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf\" (UniqueName: \"kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf\") pod \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") "
Nov 02 15:12:15.420072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.420041    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca\") pod \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\" (UID: \"84d2e9f6-a59f-4c41-ad1f-c282c1c669de\") "
Nov 02 15:12:15.428067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.416313    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca" pod "91ea7178-3cbb-4f92-bb30-597ef2c0332c" (UID: "91ea7178-3cbb-4f92-bb30-597ef2c0332c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:15.428255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.428027    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca" pod "84d2e9f6-a59f-4c41-ad1f-c282c1c669de" (UID: "84d2e9f6-a59f-4c41-ad1f-c282c1c669de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:15.453385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.452509    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca" pod "91ea7178-3cbb-4f92-bb30-597ef2c0332c" (UID: "91ea7178-3cbb-4f92-bb30-597ef2c0332c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:15.574463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.524027    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:15.575207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.575135    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea7178-3cbb-4f92-bb30-597ef2c0332c-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:15.575461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.575403    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:15.579416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.575613    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:15.662401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.653106    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf" pod "91ea7178-3cbb-4f92-bb30-597ef2c0332c" (UID: "91ea7178-3cbb-4f92-bb30-597ef2c0332c"). InnerVolumeSpecName "kube-api-access-hvbgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:15.726388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.707390    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvbgf\" (UniqueName: \"kubernetes.io/projected/91ea7178-3cbb-4f92-bb30-597ef2c0332c-kube-api-access-hvbgf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:15.726388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.715192    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf" pod "84d2e9f6-a59f-4c41-ad1f-c282c1c669de" (UID: "84d2e9f6-a59f-4c41-ad1f-c282c1c669de"). InnerVolumeSpecName "kube-api-access-wrhtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:15.790402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.776583    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca" pod "84d2e9f6-a59f-4c41-ad1f-c282c1c669de" (UID: "84d2e9f6-a59f-4c41-ad1f-c282c1c669de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:15.820401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.816959    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:15.820401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:15.817009    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhtf\" (UniqueName: \"kubernetes.io/projected/84d2e9f6-a59f-4c41-ad1f-c282c1c669de-kube-api-access-wrhtf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:16.158950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.158909    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.179794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.160543    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.213568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.160808    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.214191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.160852    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.233522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.232454    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.233522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.233256    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:12:16.236398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.235387    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:16.236398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.235448    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:16.416697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.383612    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.416697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.384400    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.416697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.384757    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:12:16.416697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.384817    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:12:16.471716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.471668    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:16.471989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.471946    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:16.717441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.639617    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.857386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:16.799830    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:12:17.218522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.101150    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.212472    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.215051    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.229609    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.231050    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.285811    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.350050    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.408668    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.387394    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.387338    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.364354    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:17.417389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.409133    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:17.706395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.698943    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:12:17.706395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.699028    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:12:17.706395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.699067    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:12:17.706395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.699105    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:12:17.738389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.715157    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:17.775753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.765427    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.852385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:17.815010    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.052695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.052587    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.144446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.128942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm\" (UniqueName: \"kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm\") pod \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") "
Nov 02 15:12:18.144446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.129032    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca\") pod \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") "
Nov 02 15:12:18.144446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.129302    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca\") pod \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\" (UID: \"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4\") "
Nov 02 15:12:18.353449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.319686    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:19.308390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:19.300720    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:19.308390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.638686    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.308390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.638651    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.308390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.392997    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:19.308390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:19.302225    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:19.308390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.688039    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.419388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.638712    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.454388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.638731    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.454388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.686733    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.454388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.404617    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca" pod "48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" (UID: "48fcfacc-200c-4487-b4dd-6e6d6eedfcb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:19.454388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.540649    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca" pod "48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" (UID: "48fcfacc-200c-4487-b4dd-6e6d6eedfcb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:19.454388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.731974    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm" pod "48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" (UID: "48fcfacc-200c-4487-b4dd-6e6d6eedfcb4"). InnerVolumeSpecName "kube-api-access-5p2pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:19.454388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.692570    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.472452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.709649    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.590327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:18.765288    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctwkf" podUID="377b7b4b-5d6b-45ed-96c2-2dd43cd58dc8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.46:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.997773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:19.059110    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.023179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:19.530053    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:20.023179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.016190    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2pm\" (UniqueName: \"kubernetes.io/projected/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-kube-api-access-5p2pm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:20.023179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.016215    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:20.023179 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:19.792115    5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.895s"
Nov 02 15:12:20.023179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.016486    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"]
Nov 02 15:12:20.075388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.069497    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg" podUID="214f8315-f258-40d4-bb11-bfae94d43ff1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.252307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:19.645828    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.252571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.252540    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:20.272487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.253110    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89" podUID="39e554bb-b67f-471a-8990-5fa031445730" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.325946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.302789    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.358514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.302843    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.358777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.333817    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:20.358931 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.358891    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.441546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.436477    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.441546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.436591    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.441546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.436637    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.591480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.591357    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.496467    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.731644    5041 cpu_manager.go:414] "containerMap: removing container" podUID="91ea7178-3cbb-4f92-bb30-597ef2c0332c" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.731671    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ea7178-3cbb-4f92-bb30-597ef2c0332c" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.731723    5041 cpu_manager.go:414] "containerMap: removing container" podUID="84d2e9f6-a59f-4c41-ad1f-c282c1c669de" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.731734    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d2e9f6-a59f-4c41-ad1f-c282c1c669de" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.731787    5041 cpu_manager.go:414] "containerMap: removing container" podUID="48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.731795    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.732011    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.732031    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ea7178-3cbb-4f92-bb30-597ef2c0332c" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.732048    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d2e9f6-a59f-4c41-ad1f-c282c1c669de" containerName="revoke-keys"
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.738080    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.741442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.738131    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:20.771595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.744653    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.771595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.744709    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:12:20.840664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.840626    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:20.882815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:20.861093    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"
Nov 02 15:12:21.177388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.161190    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.332466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.332418    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2" event={"ID":"48fcfacc-200c-4487-b4dd-6e6d6eedfcb4","Type":"ContainerDied","Data":"f288d227c3a7defd18d3e5ab0cb8a1cd4794e72bd09f6aa2ae3572167ca44ed5"}
Nov 02 15:12:21.349894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.333000    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f288d227c3a7defd18d3e5ab0cb8a1cd4794e72bd09f6aa2ae3572167ca44ed5"
Nov 02 15:12:21.349894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.333039    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"]
Nov 02 15:12:21.390719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.390684    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:12:21.427403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.427021    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:12:21.475403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.459541    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:21.475403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.459613    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:21.475403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.474345    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:12:21.587389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.584535    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vgl\" (UniqueName: \"kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.587389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.584705    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.587389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.585254    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.687069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.654399    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.740127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.740081    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.777549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.777518    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:12:21.782225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.782200    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:12:21.799799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.799762    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.800168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.800132    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vgl\" (UniqueName: \"kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.800534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.800504    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.802284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.802001    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.804259 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.804238    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:12:21.810999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.810979    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:12:21.846329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.835283    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:12:21.865708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.864784    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vgl\" (UniqueName: \"kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:21.887158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:21.887112    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324cc797-zw7mq\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:22.016397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.014771    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:22.108762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.108730    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6909-29464c"]
Nov 02 15:12:22.109161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.109126    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6909-29464c" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" containerName="test-curl-6909-29464c" containerID="cri-o://f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493" gracePeriod=2
Nov 02 15:12:22.125442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.120980    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6909-29464c"]
Nov 02 15:12:22.185151 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:22.185077    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:22.187297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.187268    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6909-1ebcf8"]
Nov 02 15:12:22.190290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.189243    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" containerName="test-curl-6909-29464c"
Nov 02 15:12:22.190290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.189266    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" containerName="test-curl-6909-29464c"
Nov 02 15:12:22.190290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.189547    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" containerName="test-curl-6909-29464c"
Nov 02 15:12:22.199472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.199443    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:22.209121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.209066    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6909-1ebcf8"]
Nov 02 15:12:22.211258 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:22.211215    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:22.236932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.236893    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:12:22.237102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.236965    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:12:22.245725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.245664    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xb7r\" (UniqueName: \"kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r\") pod \"test-curl-6909-1ebcf8\" (UID: \"0a1bd3dd-5208-49e3-bc76-d7e252e25603\") " pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:22.299394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.298840    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:22.301287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.301201    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:22.304667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.304644    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:12:22.370765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.370722    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xb7r\" (UniqueName: \"kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r\") pod \"test-curl-6909-1ebcf8\" (UID: \"0a1bd3dd-5208-49e3-bc76-d7e252e25603\") " pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:22.385146 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:22.385099    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:22.398014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.397973    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xb7r\" (UniqueName: \"kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r\") pod \"test-curl-6909-1ebcf8\" (UID: \"0a1bd3dd-5208-49e3-bc76-d7e252e25603\") " pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:22.444859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.442420    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"]
Nov 02 15:12:22.447800 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:22.447758    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:22.564160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.563851    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:22.962093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.962042    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq" event={"ID":"b64b5dc2-3b7a-4feb-861e-f3c78bba9583","Type":"ContainerStarted","Data":"9d3d283037f7ae67039bee21eded5cd71a6f34a3f1d8497ce4e55935d86bfccc"}
Nov 02 15:12:22.985962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:22.985924    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6909-1ebcf8"]
Nov 02 15:12:23.809742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.809708    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:23.810661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.810633    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:23.811793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.811769    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:12:23.837386 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:23.835752    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:23.981245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.981206    5041 generic.go:356] "Generic (PLEG): container finished" podID="b64b5dc2-3b7a-4feb-861e-f3c78bba9583" containerID="6fb3b06156b69ec63100d6cbc179db28453f01b099dc9a81e7e18fa62d84c93e" exitCode=0
Nov 02 15:12:23.981451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.981417    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq" event={"ID":"b64b5dc2-3b7a-4feb-861e-f3c78bba9583","Type":"ContainerDied","Data":"6fb3b06156b69ec63100d6cbc179db28453f01b099dc9a81e7e18fa62d84c93e"}
Nov 02 15:12:23.983260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.983212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6909-1ebcf8" event={"ID":"0a1bd3dd-5208-49e3-bc76-d7e252e25603","Type":"ContainerStarted","Data":"1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46"}
Nov 02 15:12:23.983260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:23.983255    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6909-1ebcf8" event={"ID":"0a1bd3dd-5208-49e3-bc76-d7e252e25603","Type":"ContainerStarted","Data":"5c1bd3748faa7e1e4131caa06954624aace257f62e27a3a68d369f5c374d0994"}
Nov 02 15:12:24.090555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:24.090440    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6909-1ebcf8" podStartSLOduration=2.090420148 podStartE2EDuration="2.090420148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:24.078964856 +0000 UTC m=+996.337034307" watchObservedRunningTime="2026-11-02 15:12:24.090420148 +0000 UTC m=+996.348489598"
Nov 02 15:12:24.566805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:24.557806    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:24.617146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:24.617106    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw\" (UniqueName: \"kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw\") pod \"b0c9bd54-2819-409f-8c97-4f2b8de9366f\" (UID: \"b0c9bd54-2819-409f-8c97-4f2b8de9366f\") "
Nov 02 15:12:24.652443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:24.645981    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw" pod "b0c9bd54-2819-409f-8c97-4f2b8de9366f" (UID: "b0c9bd54-2819-409f-8c97-4f2b8de9366f"). InnerVolumeSpecName "kube-api-access-j55nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:24.723769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:24.723721    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j55nw\" (UniqueName: \"kubernetes.io/projected/b0c9bd54-2819-409f-8c97-4f2b8de9366f-kube-api-access-j55nw\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:25.259449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.259324    5041 generic.go:356] "Generic (PLEG): container finished" podID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" containerID="f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493" exitCode=137
Nov 02 15:12:25.261151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.259699    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:25.261455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.259736    5041 scope.go:122] "RemoveContainer" containerID="f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493"
Nov 02 15:12:25.282773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.282729    5041 scope.go:122] "RemoveContainer" containerID="f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493"
Nov 02 15:12:25.283320 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:25.283217    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493\": container with ID starting with f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493 not found: ID does not exist" containerID="f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493"
Nov 02 15:12:25.283320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.283261    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493"} err="failed to get container status \"f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493\": rpc error: code = NotFound desc = could not find container \"f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493\": container with ID starting with f7fd5d9bb0e44a287ce437dd37f7c911c9434bb64b66382889dedbd8bfc7e493 not found: ID does not exist"
Nov 02 15:12:25.293393 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:25.293318    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:25.511844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.511815    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:25.522878 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:25.522837    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:25.580709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.580669    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6909-1ebcf8"]
Nov 02 15:12:25.587850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.587804    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca\") pod \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") "
Nov 02 15:12:25.588030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.587944    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca\") pod \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") "
Nov 02 15:12:25.588030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.588011    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl\" (UniqueName: \"kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl\") pod \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\" (UID: \"b64b5dc2-3b7a-4feb-861e-f3c78bba9583\") "
Nov 02 15:12:25.589442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.589188    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6909-1ebcf8"]
Nov 02 15:12:25.589442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.589279    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca" pod "b64b5dc2-3b7a-4feb-861e-f3c78bba9583" (UID: "b64b5dc2-3b7a-4feb-861e-f3c78bba9583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:25.592011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.591984    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca" pod "b64b5dc2-3b7a-4feb-861e-f3c78bba9583" (UID: "b64b5dc2-3b7a-4feb-861e-f3c78bba9583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:25.599003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.598969    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl" pod "b64b5dc2-3b7a-4feb-861e-f3c78bba9583" (UID: "b64b5dc2-3b7a-4feb-861e-f3c78bba9583"). InnerVolumeSpecName "kube-api-access-t4vgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.657721    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"]
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.658884    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" containerName="test-curl-6909-1ebcf8"
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.658908    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" containerName="test-curl-6909-1ebcf8"
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.658979    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b64b5dc2-3b7a-4feb-861e-f3c78bba9583" containerName="revoke-keys"
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.658989    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64b5dc2-3b7a-4feb-861e-f3c78bba9583" containerName="revoke-keys"
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.659205    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" containerName="test-curl-6909-1ebcf8"
Nov 02 15:12:25.659385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.659231    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64b5dc2-3b7a-4feb-861e-f3c78bba9583" containerName="revoke-keys"
Nov 02 15:12:25.677430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.666505    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.689387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.679170    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c2fcc8a1-openshift-default-dockercfg-7gjrk\""
Nov 02 15:12:25.689387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.685505    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"]
Nov 02 15:12:25.690282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.690259    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:25.690342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.690284    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vgl\" (UniqueName: \"kubernetes.io/projected/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-kube-api-access-t4vgl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:25.690342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.690294    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64b5dc2-3b7a-4feb-861e-f3c78bba9583-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:25.737432 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:25.735244    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-29464c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" pod="opendatahub/test-curl-6909-29464c"
Nov 02 15:12:25.792037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.791996    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792188    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792219    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792239    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792286    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792338    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792419    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792574    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjpc\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.792714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.792691    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.893322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.893279    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c9bd54-2819-409f-8c97-4f2b8de9366f" path="/var/lib/kubelet/pods/b0c9bd54-2819-409f-8c97-4f2b8de9366f/volumes"
Nov 02 15:12:25.893741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.893718    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.893803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.893767    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.893803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.893794    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjpc\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.893932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.893857    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.893987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.893949    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.894084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894067    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.894147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894103    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.894202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894177    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.894202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894190    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894494    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894498    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894510    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.894716    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.895062    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.896254    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.897269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.897198    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.916842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.916788    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:25.941426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:25.941197    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjpc\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc\") pod \"e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") " pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:26.017432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.014903    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:26.296806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.296716    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6909-1ebcf8" podUID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" containerName="test-curl-6909-1ebcf8" containerID="cri-o://1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46" gracePeriod=2
Nov 02 15:12:26.298588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.296990    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"
Nov 02 15:12:26.299933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.297028    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq" event={"ID":"b64b5dc2-3b7a-4feb-861e-f3c78bba9583","Type":"ContainerDied","Data":"9d3d283037f7ae67039bee21eded5cd71a6f34a3f1d8497ce4e55935d86bfccc"}
Nov 02 15:12:26.300099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.300082    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d3d283037f7ae67039bee21eded5cd71a6f34a3f1d8497ce4e55935d86bfccc"
Nov 02 15:12:26.781105 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:26.781069    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba06348_016b_4350_9f77_671cde29fed6.slice/crio-a9b160d5934cd34e1a18c4fa36cdb7a26450c3f72059afeb8c2046636efe220d WatchSource:0}: Error finding container a9b160d5934cd34e1a18c4fa36cdb7a26450c3f72059afeb8c2046636efe220d: Status 404 returned error can't find the container with id a9b160d5934cd34e1a18c4fa36cdb7a26450c3f72059afeb8c2046636efe220d
Nov 02 15:12:26.783044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.783016    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"]
Nov 02 15:12:26.783809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.783775    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:26.783866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.783843    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:26.783911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:26.783871    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:27.309661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:27.309623    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" event={"ID":"8ba06348-016b-4350-9f77-671cde29fed6","Type":"ContainerStarted","Data":"2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981"}
Nov 02 15:12:27.310162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:27.309687    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" event={"ID":"8ba06348-016b-4350-9f77-671cde29fed6","Type":"ContainerStarted","Data":"a9b160d5934cd34e1a18c4fa36cdb7a26450c3f72059afeb8c2046636efe220d"}
Nov 02 15:12:27.362095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:27.362044    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" podStartSLOduration=2.362025154 podStartE2EDuration="2.362025154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:27.349750165 +0000 UTC m=+999.607819615" watchObservedRunningTime="2026-11-02 15:12:27.362025154 +0000 UTC m=+999.620094608"
Nov 02 15:12:28.015885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.015847    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:28.021733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.021706    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:28.330636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.330609    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:28.331906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.331872    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:12:28.584049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.583982    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:28.668327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.668290    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r\" (UniqueName: \"kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r\") pod \"0a1bd3dd-5208-49e3-bc76-d7e252e25603\" (UID: \"0a1bd3dd-5208-49e3-bc76-d7e252e25603\") "
Nov 02 15:12:28.670515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.670476    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r" pod "0a1bd3dd-5208-49e3-bc76-d7e252e25603" (UID: "0a1bd3dd-5208-49e3-bc76-d7e252e25603"). InnerVolumeSpecName "kube-api-access-7xb7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:28.770262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.770222    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xb7r\" (UniqueName: \"kubernetes.io/projected/0a1bd3dd-5208-49e3-bc76-d7e252e25603-kube-api-access-7xb7r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:28.992900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.992865    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"]
Nov 02 15:12:28.997705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:28.997676    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.011595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.011568    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0621b336-dockercfg-w9f6k\""
Nov 02 15:12:29.014830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.014807    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0621b336\""
Nov 02 15:12:29.019024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.018999    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"]
Nov 02 15:12:29.078894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.074873    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.240005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.079963    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wq4\" (UniqueName: \"kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4\") pod \"payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.377559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.349697    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"]
Nov 02 15:12:29.377559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.361352    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.381898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.380920    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.381898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.381071    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wq4\" (UniqueName: \"kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4\") pod \"payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.382500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.382474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.387316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.387287    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"]
Nov 02 15:12:29.396393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.395165    5041 generic.go:356] "Generic (PLEG): container finished" podID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" containerID="1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46" exitCode=137
Nov 02 15:12:29.396393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.396322    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:29.404428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.404403    5041 scope.go:122] "RemoveContainer" containerID="1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46"
Nov 02 15:12:29.466490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.459182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wq4\" (UniqueName: \"kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4\") pod \"payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.468686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.468661    5041 scope.go:122] "RemoveContainer" containerID="1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46"
Nov 02 15:12:29.469025 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:29.468999    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46\": container with ID starting with 1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46 not found: ID does not exist" containerID="1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46"
Nov 02 15:12:29.469082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.469036    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46"} err="failed to get container status \"1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46\": rpc error: code = NotFound desc = could not find container \"1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46\": container with ID starting with 1db1cf8a6c2e66b520967a033e21eec60cc874d0a895cba506d39e28b9196c46 not found: ID does not exist"
Nov 02 15:12:29.483319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.483295    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hztj\" (UniqueName: \"kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj\") pod \"payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.483456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.483404    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.487691 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:29.487656    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6909-1ebcf8\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" pod="opendatahub/test-curl-6909-1ebcf8"
Nov 02 15:12:29.585338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.585296    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hztj\" (UniqueName: \"kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj\") pod \"payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.585523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.585418    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.586149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.586126    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.613808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.613777    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:29.614052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.614031    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hztj\" (UniqueName: \"kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj\") pod \"payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.697443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.697396    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:29.892437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.892353    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1bd3dd-5208-49e3-bc76-d7e252e25603" path="/var/lib/kubelet/pods/0a1bd3dd-5208-49e3-bc76-d7e252e25603/volumes"
Nov 02 15:12:29.947147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:29.946164    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"]
Nov 02 15:12:30.035527 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:30.035500    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d0b8ed_a61e_4176_899c_356a60c31c16.slice/crio-ffccdc16d41ea667e0de711af7b567c70f8a9a4c4ac47bbc3ad7e77c4165c211 WatchSource:0}: Error finding container ffccdc16d41ea667e0de711af7b567c70f8a9a4c4ac47bbc3ad7e77c4165c211: Status 404 returned error can't find the container with id ffccdc16d41ea667e0de711af7b567c70f8a9a4c4ac47bbc3ad7e77c4165c211
Nov 02 15:12:30.038585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.038561    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"]
Nov 02 15:12:30.406222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.406183    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5" event={"ID":"badf915b-c1e2-46b9-8c41-80bef554df20","Type":"ContainerStarted","Data":"7f8c7f68133d7161e4b0bb5a9e7b5cb4c5aed5af512605fa83be992025f8d4b2"}
Nov 02 15:12:30.406735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.406254    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5" event={"ID":"badf915b-c1e2-46b9-8c41-80bef554df20","Type":"ContainerStarted","Data":"80714de60e2c54b1a0c7b2fafc90867f229b949405af122d9a24613f77a75bc1"}
Nov 02 15:12:30.406735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.406284    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:30.409446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.409408    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" event={"ID":"d0d0b8ed-a61e-4176-899c-356a60c31c16","Type":"ContainerStarted","Data":"26e9a85bb9f15f382de1b75fe747445c0aa012497801264b08a0603fbcb7efbb"}
Nov 02 15:12:30.409615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.409587    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:30.409738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.409642    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" event={"ID":"d0d0b8ed-a61e-4176-899c-356a60c31c16","Type":"ContainerStarted","Data":"ffccdc16d41ea667e0de711af7b567c70f8a9a4c4ac47bbc3ad7e77c4165c211"}
Nov 02 15:12:30.455330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.455283    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5" podStartSLOduration=2.455271584 podStartE2EDuration="2.455271584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:30.447049897 +0000 UTC m=+1002.705119347" watchObservedRunningTime="2026-11-02 15:12:30.455271584 +0000 UTC m=+1002.713341034"
Nov 02 15:12:30.495483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:30.495440    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" podStartSLOduration=1.495427938 podStartE2EDuration="1.495427938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:30.489797988 +0000 UTC m=+1002.747867463" watchObservedRunningTime="2026-11-02 15:12:30.495427938 +0000 UTC m=+1002.753497401"
Nov 02 15:12:31.617760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.617726    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"]
Nov 02 15:12:31.623410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.623358    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:31.643732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.643700    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"]
Nov 02 15:12:31.645528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.645505    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c2fcc8a1\""
Nov 02 15:12:31.678456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.678428    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"]
Nov 02 15:12:31.683885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.683862    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.700969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.700941    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c2fcc8a1-dockercfg-8sdd5\""
Nov 02 15:12:31.712605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.712570    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"]
Nov 02 15:12:31.723198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.723172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdv85\" (UniqueName: \"kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85\") pod \"payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") " pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:31.723543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.723518    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") " pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:31.723660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.723603    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.723749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.723728    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxmr\" (UniqueName: \"kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr\") pod \"payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.827340    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.827516    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxmr\" (UniqueName: \"kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr\") pod \"payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.827606    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdv85\" (UniqueName: \"kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85\") pod \"payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") " pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.827838    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") " pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.828871    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") " pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.835648    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.891598    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxmr\" (UniqueName: \"kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr\") pod \"payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:31.940778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:31.896128    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdv85\" (UniqueName: \"kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85\") pod \"payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") " pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:32.050385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.050320    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:32.103387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.099704    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:32.145643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.142156    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"]
Nov 02 15:12:32.171350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.169897    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"]
Nov 02 15:12:32.171350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.170020    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.183524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.182838    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:12:32.183524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.182933    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:32.253864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.253821    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d6z\" (UniqueName: \"kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.253992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.253974    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.254151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.254131    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.355804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.355772    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d6z\" (UniqueName: \"kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.355959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.355838    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.355959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.355888    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.356702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.356668    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.358346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.358323    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.382497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.382472    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d6z\" (UniqueName: \"kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z\") pod \"maas-api-revoke-sub-2a9d4011-s6s8c\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.486566 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:32.486496    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfbe668_8d57_415d_a41d_003c050128a8.slice/crio-f11cf1388a02041804c25c16c997bd0b47c545e782fd4a07580119d24e2e8317 WatchSource:0}: Error finding container f11cf1388a02041804c25c16c997bd0b47c545e782fd4a07580119d24e2e8317: Status 404 returned error can't find the container with id f11cf1388a02041804c25c16c997bd0b47c545e782fd4a07580119d24e2e8317
Nov 02 15:12:32.489070 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.489046    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"]
Nov 02 15:12:32.499072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.499052    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"]
Nov 02 15:12:32.501583 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:32.501558    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebdd3dc_d531_486f_9e99_178f94da21d3.slice/crio-037b139b78a805ccdf5fdf03f0e85ab90af1dd6cbd8e5806aa5a015cf90b05ec WatchSource:0}: Error finding container 037b139b78a805ccdf5fdf03f0e85ab90af1dd6cbd8e5806aa5a015cf90b05ec: Status 404 returned error can't find the container with id 037b139b78a805ccdf5fdf03f0e85ab90af1dd6cbd8e5806aa5a015cf90b05ec
Nov 02 15:12:32.564130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.564101    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:32.900026 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:32.899992    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1384b9b_ac30_4834_b37d_b013db4b294f.slice/crio-2f3c64251adffb0194035c4c6d4ae2d3a95d5a73051178b8d5306118677fd776 WatchSource:0}: Error finding container 2f3c64251adffb0194035c4c6d4ae2d3a95d5a73051178b8d5306118677fd776: Status 404 returned error can't find the container with id 2f3c64251adffb0194035c4c6d4ae2d3a95d5a73051178b8d5306118677fd776
Nov 02 15:12:32.902805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:32.902781    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"]
Nov 02 15:12:33.445315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.445275    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c" event={"ID":"b1384b9b-ac30-4834-b37d-b013db4b294f","Type":"ContainerStarted","Data":"11756cfa99147cf9d911048ee3436fb0693fbd3ec4961596ddc8fabce40fd44f"}
Nov 02 15:12:33.445500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.445344    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c" event={"ID":"b1384b9b-ac30-4834-b37d-b013db4b294f","Type":"ContainerStarted","Data":"2f3c64251adffb0194035c4c6d4ae2d3a95d5a73051178b8d5306118677fd776"}
Nov 02 15:12:33.447116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.447085    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" event={"ID":"6ebdd3dc-d531-486f-9e99-178f94da21d3","Type":"ContainerStarted","Data":"b321eceb525b7eadaa5b3097a46381d1c0bfc650c8c820ac98ed3e9ad04546c3"}
Nov 02 15:12:33.447238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.447130    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" event={"ID":"6ebdd3dc-d531-486f-9e99-178f94da21d3","Type":"ContainerStarted","Data":"037b139b78a805ccdf5fdf03f0e85ab90af1dd6cbd8e5806aa5a015cf90b05ec"}
Nov 02 15:12:33.447238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.447188    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:33.448933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.448908    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" event={"ID":"ddfbe668-8d57-415d-a41d-003c050128a8","Type":"ContainerStarted","Data":"41efe1a79665d61956e52fc00867abc61518be7b2e8de85db705a15b410c9973"}
Nov 02 15:12:33.449044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.448954    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:33.449044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.448975    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" event={"ID":"ddfbe668-8d57-415d-a41d-003c050128a8","Type":"ContainerStarted","Data":"f11cf1388a02041804c25c16c997bd0b47c545e782fd4a07580119d24e2e8317"}
Nov 02 15:12:33.486400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.486306    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c" podStartSLOduration=1.486292876 podStartE2EDuration="1.486292876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:33.480467762 +0000 UTC m=+1005.738537212" watchObservedRunningTime="2026-11-02 15:12:33.486292876 +0000 UTC m=+1005.744362329"
Nov 02 15:12:33.517329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.517274    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" podStartSLOduration=2.517256331 podStartE2EDuration="2.517256331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:33.515589573 +0000 UTC m=+1005.773659024" watchObservedRunningTime="2026-11-02 15:12:33.517256331 +0000 UTC m=+1005.775325784"
Nov 02 15:12:33.559443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:33.559405    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" podStartSLOduration=2.559392377 podStartE2EDuration="2.559392377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:33.554987574 +0000 UTC m=+1005.813057025" watchObservedRunningTime="2026-11-02 15:12:33.559392377 +0000 UTC m=+1005.817461832"
Nov 02 15:12:34.549440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:34.549387    5041 generic.go:356] "Generic (PLEG): container finished" podID="b1384b9b-ac30-4834-b37d-b013db4b294f" containerID="11756cfa99147cf9d911048ee3436fb0693fbd3ec4961596ddc8fabce40fd44f" exitCode=0
Nov 02 15:12:34.550528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:34.550491    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c" event={"ID":"b1384b9b-ac30-4834-b37d-b013db4b294f","Type":"ContainerDied","Data":"11756cfa99147cf9d911048ee3436fb0693fbd3ec4961596ddc8fabce40fd44f"}
Nov 02 15:12:35.727974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.727940    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:35.849009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.848924    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca\") pod \"b1384b9b-ac30-4834-b37d-b013db4b294f\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") "
Nov 02 15:12:35.849153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.849105    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z\" (UniqueName: \"kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z\") pod \"b1384b9b-ac30-4834-b37d-b013db4b294f\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") "
Nov 02 15:12:35.849306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.849279    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca" pod "b1384b9b-ac30-4834-b37d-b013db4b294f" (UID: "b1384b9b-ac30-4834-b37d-b013db4b294f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:35.849419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.849289    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca\") pod \"b1384b9b-ac30-4834-b37d-b013db4b294f\" (UID: \"b1384b9b-ac30-4834-b37d-b013db4b294f\") "
Nov 02 15:12:35.850048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.850022    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:35.851408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.851349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z" pod "b1384b9b-ac30-4834-b37d-b013db4b294f" (UID: "b1384b9b-ac30-4834-b37d-b013db4b294f"). InnerVolumeSpecName "kube-api-access-k6d6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:35.851760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.851732    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca" pod "b1384b9b-ac30-4834-b37d-b013db4b294f" (UID: "b1384b9b-ac30-4834-b37d-b013db4b294f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:35.951793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.951759    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6d6z\" (UniqueName: \"kubernetes.io/projected/b1384b9b-ac30-4834-b37d-b013db4b294f-kube-api-access-k6d6z\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:35.951793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:35.951789    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1384b9b-ac30-4834-b37d-b013db4b294f-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:36.572358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:36.572324    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"
Nov 02 15:12:36.572358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:36.572336    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c" event={"ID":"b1384b9b-ac30-4834-b37d-b013db4b294f","Type":"ContainerDied","Data":"2f3c64251adffb0194035c4c6d4ae2d3a95d5a73051178b8d5306118677fd776"}
Nov 02 15:12:36.572571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:36.572424    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3c64251adffb0194035c4c6d4ae2d3a95d5a73051178b8d5306118677fd776"
Nov 02 15:12:36.770200 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:36.770164    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:12:39.456612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.456573    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"]
Nov 02 15:12:39.457391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.457342    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b1384b9b-ac30-4834-b37d-b013db4b294f" containerName="revoke-keys"
Nov 02 15:12:39.457391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.457375    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1384b9b-ac30-4834-b37d-b013db4b294f" containerName="revoke-keys"
Nov 02 15:12:39.457595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.457583    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1384b9b-ac30-4834-b37d-b013db4b294f" containerName="revoke-keys"
Nov 02 15:12:39.463542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.463521    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.509417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.509382    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"]
Nov 02 15:12:39.540405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540343    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwqj\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.540559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.540717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540691    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.540791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540773    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.540839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540813    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.540882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540866    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.540939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.540922    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.541051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.541025    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.541127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.541115    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642119    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwqj\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642251    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642329    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642408    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642446    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642486    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642539    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642548    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642590    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642834    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642871    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.642984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.642924    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.643659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.643625    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.645220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.645192    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.645318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.645216    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.683240 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.683209    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwqj\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.686985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.686815    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.779198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.779114    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:39.791717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.791684    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"]
Nov 02 15:12:39.797342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.797323    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:39.818646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.818558    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-649ad0be\""
Nov 02 15:12:39.821547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.821518    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"]
Nov 02 15:12:39.888433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.877454    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"]
Nov 02 15:12:39.888433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.885664    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:39.919947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:39.919890    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"]
Nov 02 15:12:40.032562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.023205    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-649ad0be-dockercfg-rl9m8\""
Nov 02 15:12:40.143387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.133313    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6frg\" (UniqueName: \"kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg\") pod \"payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.151386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.149455    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xz9\" (UniqueName: \"kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9\") pod \"payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") " pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.151386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.149767    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.151386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.149934    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume\") pod \"payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") " pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.221392 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:40.220666    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf478ae8_7689_466f_8158_8edf00a0a2cb.slice/crio-da13df3c34de7f24f8c66d7b0e0adc9b150be4f99be01835a66c044de335d303 WatchSource:0}: Error finding container da13df3c34de7f24f8c66d7b0e0adc9b150be4f99be01835a66c044de335d303: Status 404 returned error can't find the container with id da13df3c34de7f24f8c66d7b0e0adc9b150be4f99be01835a66c044de335d303
Nov 02 15:12:40.255645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.255243    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:40.256649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.256512    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:40.277519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.277456    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:40.279004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.258343    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6frg\" (UniqueName: \"kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg\") pod \"payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.279270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.279249    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xz9\" (UniqueName: \"kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9\") pod \"payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") " pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.280766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.279898    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.281121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.281076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume\") pod \"payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") " pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.282769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.282695    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume\") pod \"payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") " pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.284108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.284077    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.303495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.299759    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"]
Nov 02 15:12:40.345473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.342590    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6frg\" (UniqueName: \"kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg\") pod \"payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.350021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.348852    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xz9\" (UniqueName: \"kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9\") pod \"payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") " pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.459265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.459230    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:40.532719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.532644    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:40.638227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.636888    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" event={"ID":"df478ae8-7689-466f-8158-8edf00a0a2cb","Type":"ContainerStarted","Data":"aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c"}
Nov 02 15:12:40.638227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.636959    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" event={"ID":"df478ae8-7689-466f-8158-8edf00a0a2cb","Type":"ContainerStarted","Data":"da13df3c34de7f24f8c66d7b0e0adc9b150be4f99be01835a66c044de335d303"}
Nov 02 15:12:40.703582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.703514    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podStartSLOduration=1.703490725 podStartE2EDuration="1.703490725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:40.69779629 +0000 UTC m=+1012.955865753" watchObservedRunningTime="2026-11-02 15:12:40.703490725 +0000 UTC m=+1012.961560176"
Nov 02 15:12:40.779594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.779560    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:40.797838 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:40.797801    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d366a77_f87a_4111_8166_9fa955d2e681.slice/crio-2beaaa989dd3e2e2458e311b6acf9744b73e4c4cacc1cee406c08eb7d018c210 WatchSource:0}: Error finding container 2beaaa989dd3e2e2458e311b6acf9744b73e4c4cacc1cee406c08eb7d018c210: Status 404 returned error can't find the container with id 2beaaa989dd3e2e2458e311b6acf9744b73e4c4cacc1cee406c08eb7d018c210
Nov 02 15:12:40.798749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.798723    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"]
Nov 02 15:12:40.955717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:40.955689    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"]
Nov 02 15:12:41.444913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.444880    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:12:41.445708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.445676    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:12:41.670713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.670676    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" event={"ID":"e93e760f-6eec-46e3-824c-2708f9df02f4","Type":"ContainerStarted","Data":"b5055aeed3d9a4aeb3cc6a17277954e0614dddceb619d647958322db21b41c6a"}
Nov 02 15:12:41.671226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.670734    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" event={"ID":"e93e760f-6eec-46e3-824c-2708f9df02f4","Type":"ContainerStarted","Data":"e8f47d8c30369c4c28e2b4d35db969cb8d4f2f2192e81e955e86a20e7ed77e7a"}
Nov 02 15:12:41.671226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.670795    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:41.672800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.672772    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh" event={"ID":"8d366a77-f87a-4111-8166-9fa955d2e681","Type":"ContainerStarted","Data":"ec9d681f01d0b8617f9e9c6f14d0e3ee1690630a3344cb15cb3d41d238307443"}
Nov 02 15:12:41.672924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.672827    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:41.672924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.672845    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh" event={"ID":"8d366a77-f87a-4111-8166-9fa955d2e681","Type":"ContainerStarted","Data":"2beaaa989dd3e2e2458e311b6acf9744b73e4c4cacc1cee406c08eb7d018c210"}
Nov 02 15:12:41.736445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.725824    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" podStartSLOduration=2.725801751 podStartE2EDuration="2.725801751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:41.710959273 +0000 UTC m=+1013.969028723" watchObservedRunningTime="2026-11-02 15:12:41.725801751 +0000 UTC m=+1013.983871203"
Nov 02 15:12:41.770757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.770704    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh" podStartSLOduration=2.770689915 podStartE2EDuration="2.770689915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:41.767534418 +0000 UTC m=+1014.025603869" watchObservedRunningTime="2026-11-02 15:12:41.770689915 +0000 UTC m=+1014.028759365"
Nov 02 15:12:41.781935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.781901    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:41.782111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:41.781955    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:42.745102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.745065    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"]
Nov 02 15:12:42.750929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.750903    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:42.762555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.762530    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"]
Nov 02 15:12:42.780161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.779808    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d3bd1bf1-dockercfg-vjm28\""
Nov 02 15:12:42.780161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.779886    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d3bd1bf1\""
Nov 02 15:12:42.804532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.782547    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:42.804532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:42.782630    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:43.254257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.254209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qd8\" (UniqueName: \"kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8\") pod \"payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.254725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.254701    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.290555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.290523    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"]
Nov 02 15:12:43.295141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.295112    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.297680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.297651    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"]
Nov 02 15:12:43.308612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.304697    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"]
Nov 02 15:12:43.310049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.310026    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.327388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.322344    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:43.327388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.322548    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:12:43.348810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.347480    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"]
Nov 02 15:12:43.357029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.356998    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.357182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357062    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg55n\" (UniqueName: \"kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.357182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357089    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.357182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357131    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj46f\" (UniqueName: \"kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f\") pod \"payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") " pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.357294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357246    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") " pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.357333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qd8\" (UniqueName: \"kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8\") pod \"payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.357408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357353    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.357824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.357805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.379175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.379144    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qd8\" (UniqueName: \"kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8\") pod \"payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.458619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.458583    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.458847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.458826    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg55n\" (UniqueName: \"kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.458932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.458877    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.458995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.458932    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj46f\" (UniqueName: \"kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f\") pod \"payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") " pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.459067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.459056    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") " pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.459376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.459336    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.459856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.459834    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") " pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.461279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.461260    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.479662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.479624    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj46f\" (UniqueName: \"kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f\") pod \"payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") " pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.479889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.479865    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg55n\" (UniqueName: \"kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n\") pod \"maas-api-revoke-sub-e1030fd3-ngqf4\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.622438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.622171    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:43.642455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.636636    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:43.678897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.676708    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:43.781173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.781135    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:43.781769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:43.781736    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:44.147126 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:44.147089    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcc045a_4708_4c3e_9b55_f3181a7dfed2.slice/crio-94c972154a8c01faf4f5114f2e0bccee6e3e0dda15f6326ad48a9080f6164425 WatchSource:0}: Error finding container 94c972154a8c01faf4f5114f2e0bccee6e3e0dda15f6326ad48a9080f6164425: Status 404 returned error can't find the container with id 94c972154a8c01faf4f5114f2e0bccee6e3e0dda15f6326ad48a9080f6164425
Nov 02 15:12:44.147756 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:44.147722    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc21ddd_b54b_4848_ba15_1779ea2ebee0.slice/crio-9bed5597e093c5234d95eb2b07fb326eed772a3d0f9c72d899d0f628994e80a1 WatchSource:0}: Error finding container 9bed5597e093c5234d95eb2b07fb326eed772a3d0f9c72d899d0f628994e80a1: Status 404 returned error can't find the container with id 9bed5597e093c5234d95eb2b07fb326eed772a3d0f9c72d899d0f628994e80a1
Nov 02 15:12:44.150966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.150946    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"]
Nov 02 15:12:44.159099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.159062    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"]
Nov 02 15:12:44.175320 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:12:44.174555    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53742354_8c0c_46bc_84e2_ab9bde2c321d.slice/crio-5b34b7e2535fa20cd41014223072b1ede6419a93220bbc07d2e371f9c6454ff6 WatchSource:0}: Error finding container 5b34b7e2535fa20cd41014223072b1ede6419a93220bbc07d2e371f9c6454ff6: Status 404 returned error can't find the container with id 5b34b7e2535fa20cd41014223072b1ede6419a93220bbc07d2e371f9c6454ff6
Nov 02 15:12:44.178113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.178081    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"]
Nov 02 15:12:44.555578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.555128    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:12:44.556770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.555970    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:12:44.779804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.779768    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:44.779974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.779825    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:44.865448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.859791    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn" event={"ID":"bdcc045a-4708-4c3e-9b55-f3181a7dfed2","Type":"ContainerStarted","Data":"d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33"}
Nov 02 15:12:44.865448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.859866    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn" event={"ID":"bdcc045a-4708-4c3e-9b55-f3181a7dfed2","Type":"ContainerStarted","Data":"94c972154a8c01faf4f5114f2e0bccee6e3e0dda15f6326ad48a9080f6164425"}
Nov 02 15:12:44.865448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.861020    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:44.865448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.864126    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx" event={"ID":"53742354-8c0c-46bc-84e2-ab9bde2c321d","Type":"ContainerStarted","Data":"5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e"}
Nov 02 15:12:44.865448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.864170    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx" event={"ID":"53742354-8c0c-46bc-84e2-ab9bde2c321d","Type":"ContainerStarted","Data":"5b34b7e2535fa20cd41014223072b1ede6419a93220bbc07d2e371f9c6454ff6"}
Nov 02 15:12:44.865448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.864781    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:44.867138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.867111    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4" event={"ID":"8bc21ddd-b54b-4848-ba15-1779ea2ebee0","Type":"ContainerStarted","Data":"02d8db49a6e88380dd67cc651d9502f5394a874ee9be11d90b5fbfc927abfacf"}
Nov 02 15:12:44.867274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.867154    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4" event={"ID":"8bc21ddd-b54b-4848-ba15-1779ea2ebee0","Type":"ContainerStarted","Data":"9bed5597e093c5234d95eb2b07fb326eed772a3d0f9c72d899d0f628994e80a1"}
Nov 02 15:12:44.904524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.904476    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn" podStartSLOduration=2.904461916 podStartE2EDuration="2.904461916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:44.902549043 +0000 UTC m=+1017.160618494" watchObservedRunningTime="2026-11-02 15:12:44.904461916 +0000 UTC m=+1017.162531365"
Nov 02 15:12:44.942887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.942812    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4" podStartSLOduration=1.942793616 podStartE2EDuration="1.942793616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:44.939989938 +0000 UTC m=+1017.198059391" watchObservedRunningTime="2026-11-02 15:12:44.942793616 +0000 UTC m=+1017.200863067"
Nov 02 15:12:44.984219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:44.984152    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx" podStartSLOduration=2.984133802 podStartE2EDuration="2.984133802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:44.97470916 +0000 UTC m=+1017.232778636" watchObservedRunningTime="2026-11-02 15:12:44.984133802 +0000 UTC m=+1017.242203253"
Nov 02 15:12:45.381094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.381057    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"]
Nov 02 15:12:45.385210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.385187    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.405158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.405109    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0621b336-serving-cert\""
Nov 02 15:12:45.409475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.409433    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"]
Nov 02 15:12:45.457706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.457661    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8j4g\" (UniqueName: \"kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.458714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.458108    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.458978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.458959    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.459192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.459175    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.599440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.594870    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8j4g\" (UniqueName: \"kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.599440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.595054    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.609441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.605491    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.609441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.605628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.609441 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:45.606065    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-serving-cert: secret "maas-api-e2e-migrate-0621b336-serving-cert" not found
Nov 02 15:12:45.609441 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:12:45.606148    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls podName:390dccfe-5b8e-4be8-a109-1beeead39593 nodeName:}" failed. No retries permitted until 2026-11-02 15:12:46.106125621 +0000 UTC m=+1018.364195070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls") pod "maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" (UID: "390dccfe-5b8e-4be8-a109-1beeead39593") : secret "maas-api-e2e-migrate-0621b336-serving-cert" not found
Nov 02 15:12:45.613459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.610951    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.641400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.640615    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8j4g\" (UniqueName: \"kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.699395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.697272    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:45.789460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.784789    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:45.789460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.784851    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:45.880841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.880762    5041 generic.go:356] "Generic (PLEG): container finished" podID="8bc21ddd-b54b-4848-ba15-1779ea2ebee0" containerID="02d8db49a6e88380dd67cc651d9502f5394a874ee9be11d90b5fbfc927abfacf" exitCode=0
Nov 02 15:12:45.890195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:45.890163    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4" event={"ID":"8bc21ddd-b54b-4848-ba15-1779ea2ebee0","Type":"ContainerDied","Data":"02d8db49a6e88380dd67cc651d9502f5394a874ee9be11d90b5fbfc927abfacf"}
Nov 02 15:12:46.148877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:46.148792    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:46.151308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:46.151285    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\") pod \"maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:46.395387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:46.392711    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:46.810469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:46.804744    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:46.810469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:46.804813    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:46.950106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:46.950052    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"]
Nov 02 15:12:47.165514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.165487    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:47.273650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.273614    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n\" (UniqueName: \"kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n\") pod \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") "
Nov 02 15:12:47.273834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.273668    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca\") pod \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") "
Nov 02 15:12:47.273834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.273718    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca\") pod \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\" (UID: \"8bc21ddd-b54b-4848-ba15-1779ea2ebee0\") "
Nov 02 15:12:47.274155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.274109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca" pod "8bc21ddd-b54b-4848-ba15-1779ea2ebee0" (UID: "8bc21ddd-b54b-4848-ba15-1779ea2ebee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:47.274685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.274655    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:47.275936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.275907    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca" pod "8bc21ddd-b54b-4848-ba15-1779ea2ebee0" (UID: "8bc21ddd-b54b-4848-ba15-1779ea2ebee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:47.276054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.276033    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n" pod "8bc21ddd-b54b-4848-ba15-1779ea2ebee0" (UID: "8bc21ddd-b54b-4848-ba15-1779ea2ebee0"). InnerVolumeSpecName "kube-api-access-xg55n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:47.356852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.356809    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"]
Nov 02 15:12:47.357667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.357644    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8bc21ddd-b54b-4848-ba15-1779ea2ebee0" containerName="revoke-keys"
Nov 02 15:12:47.357667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.357662    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc21ddd-b54b-4848-ba15-1779ea2ebee0" containerName="revoke-keys"
Nov 02 15:12:47.357855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.357834    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc21ddd-b54b-4848-ba15-1779ea2ebee0" containerName="revoke-keys"
Nov 02 15:12:47.361243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.361218    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.380159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.376621    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg55n\" (UniqueName: \"kubernetes.io/projected/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-kube-api-access-xg55n\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:47.380406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.380384    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc21ddd-b54b-4848-ba15-1779ea2ebee0-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:12:47.382208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.382157    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c2fcc8a1-serving-cert\""
Nov 02 15:12:47.388731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.388704    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"]
Nov 02 15:12:47.481761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.481729    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bsr\" (UniqueName: \"kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.481894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.481781    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.481986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.481969    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.482151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.482135    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.585544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.584868    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bsr\" (UniqueName: \"kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.588533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.588499    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.590144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.590119    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.591011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.590987    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.591792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.589928    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.641781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.639387    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.662008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.661118    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.762422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.756045    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bsr\" (UniqueName: \"kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr\") pod \"maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.793386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.793038    5041 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 15:12:47.793386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.793127    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 15:12:47.981907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.981868    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:47.984600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.984522    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"
Nov 02 15:12:47.984600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.984545    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4" event={"ID":"8bc21ddd-b54b-4848-ba15-1779ea2ebee0","Type":"ContainerDied","Data":"9bed5597e093c5234d95eb2b07fb326eed772a3d0f9c72d899d0f628994e80a1"}
Nov 02 15:12:47.985071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.985048    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bed5597e093c5234d95eb2b07fb326eed772a3d0f9c72d899d0f628994e80a1"
Nov 02 15:12:47.987480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.987449    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" event={"ID":"390dccfe-5b8e-4be8-a109-1beeead39593","Type":"ContainerStarted","Data":"cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b"}
Nov 02 15:12:47.987677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.987656    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" event={"ID":"390dccfe-5b8e-4be8-a109-1beeead39593","Type":"ContainerStarted","Data":"744869935d2b3abecc5b91906d889fe4c2e7ab7df9031a2846e7456f2e3e6969"}
Nov 02 15:12:47.987885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:47.987871    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:48.036386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:48.036320    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" podStartSLOduration=2.521893306 podStartE2EDuration="3.036304604s" totalImagesPullingTime="514.411298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:48.031221219 +0000 UTC m=+1020.289290681" watchObservedRunningTime="2026-11-02 15:12:48.036304604 +0000 UTC m=+1020.294374054"
Nov 02 15:12:48.384934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:48.384902    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"]
Nov 02 15:12:48.784929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:48.784897    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:48.785426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:48.785399    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:48.786087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:48.786066    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:12:49.110426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:49.110388    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw" event={"ID":"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a","Type":"ContainerStarted","Data":"5d99b41a318a54e21591576be3068d280a9611f29e84f6ba031388e88be5b905"}
Nov 02 15:12:50.121855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:50.121804    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw" event={"ID":"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a","Type":"ContainerStarted","Data":"4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5"}
Nov 02 15:12:50.122257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:50.121879    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:50.159875 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:50.159827    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw" podStartSLOduration=2.458827205 podStartE2EDuration="3.159813288s" totalImagesPullingTime="700.986083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:50.158692632 +0000 UTC m=+1022.416762083" watchObservedRunningTime="2026-11-02 15:12:50.159813288 +0000 UTC m=+1022.417882761"
Nov 02 15:12:52.699388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:52.698635    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:12:52.705384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:52.702306    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:12:54.169476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:54.141837    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:12:56.166519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:56.146090    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:12:56.969828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:56.969800    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:12:56.970008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:56.969870    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:12:59.348660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.348580    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"]
Nov 02 15:12:59.353437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.353420    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.367344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.367315    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-649ad0be-serving-cert\""
Nov 02 15:12:59.376106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.376077    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"]
Nov 02 15:12:59.479907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.479873    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.480048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.480018    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.480096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.480082    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2vr\" (UniqueName: \"kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.480158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.480148    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.584388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.583493    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.584388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.583667    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.584388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.583729    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2vr\" (UniqueName: \"kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.584388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.583833    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.603514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.595393    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.695150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.693329    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.695150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.693561    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.708508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.703951    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2vr\" (UniqueName: \"kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr\") pod \"maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:12:59.969610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:12:59.969577    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:13:00.284098 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:00.284069    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079839b5_efef_4707_bab1_15156d3a2275.slice/crio-359e18c1419f35a88ae5657b35fac5c68072f045273595dd8f65f2dd5e2545b1 WatchSource:0}: Error finding container 359e18c1419f35a88ae5657b35fac5c68072f045273595dd8f65f2dd5e2545b1: Status 404 returned error can't find the container with id 359e18c1419f35a88ae5657b35fac5c68072f045273595dd8f65f2dd5e2545b1
Nov 02 15:13:00.285958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:00.285933    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:13:00.287623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:00.287597    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"]
Nov 02 15:13:00.813980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:00.813938    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7" event={"ID":"079839b5-efef-4707-bab1-15156d3a2275","Type":"ContainerStarted","Data":"359e18c1419f35a88ae5657b35fac5c68072f045273595dd8f65f2dd5e2545b1"}
Nov 02 15:13:01.827733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:01.827691    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7" event={"ID":"079839b5-efef-4707-bab1-15156d3a2275","Type":"ContainerStarted","Data":"c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46"}
Nov 02 15:13:01.828165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:01.827797    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:13:01.868473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:01.868426    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7" podStartSLOduration=2.298869014 podStartE2EDuration="2.86841331s" totalImagesPullingTime="569.544296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:01.866611947 +0000 UTC m=+1034.124681398" watchObservedRunningTime="2026-11-02 15:13:01.86841331 +0000 UTC m=+1034.126482762"
Nov 02 15:13:03.049038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.049001    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"]
Nov 02 15:13:03.053311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.053291    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.069265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.069204    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d3bd1bf1-serving-cert\""
Nov 02 15:13:03.088669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.088641    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"]
Nov 02 15:13:03.187127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.187096    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.187270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.187243    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.187315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.187300    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.187396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.187372    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fsm\" (UniqueName: \"kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.322462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.303138    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.322462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.303306    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.322462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.303413    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.322462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.303466    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fsm\" (UniqueName: \"kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.323902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.323867    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.339675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.339634    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.365389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.354092    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fsm\" (UniqueName: \"kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.365389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.354671    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs\") pod \"maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.386383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.381796    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:03.797564 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:03.797534    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075a0fbc_afa6_47f3_932f_93de6743307e.slice/crio-1361df79225063103d6df9bd25573b777d02a14643027db246d9212c4231bd66 WatchSource:0}: Error finding container 1361df79225063103d6df9bd25573b777d02a14643027db246d9212c4231bd66: Status 404 returned error can't find the container with id 1361df79225063103d6df9bd25573b777d02a14643027db246d9212c4231bd66
Nov 02 15:13:03.801133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.801108    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"]
Nov 02 15:13:03.953820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:03.953772    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx" event={"ID":"075a0fbc-afa6-47f3-932f-93de6743307e","Type":"ContainerStarted","Data":"1361df79225063103d6df9bd25573b777d02a14643027db246d9212c4231bd66"}
Nov 02 15:13:04.047597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:04.047498    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"]
Nov 02 15:13:04.053738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:04.053705    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b06ea6-tmpqz"]
Nov 02 15:13:04.966233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:04.966191    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx" event={"ID":"075a0fbc-afa6-47f3-932f-93de6743307e","Type":"ContainerStarted","Data":"24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c"}
Nov 02 15:13:04.966428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:04.966261    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:05.014394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:05.014304    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx" podStartSLOduration=1.440472371 podStartE2EDuration="2.014281549s" totalImagesPullingTime="573.809178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:05.012942492 +0000 UTC m=+1037.271011949" watchObservedRunningTime="2026-11-02 15:13:05.014281549 +0000 UTC m=+1037.272351001"
Nov 02 15:13:05.096613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:05.096541    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"]
Nov 02 15:13:05.123061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:05.122974    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649a2b12-zn2xg"]
Nov 02 15:13:05.891266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:05.891232    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501afba0-6ba3-4977-bc41-aadc514aa6d6" path="/var/lib/kubelet/pods/501afba0-6ba3-4977-bc41-aadc514aa6d6/volumes"
Nov 02 15:13:05.891911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:05.891889    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af" path="/var/lib/kubelet/pods/9c62b5a7-f8e0-4c03-9970-b1aa07f4d9af/volumes"
Nov 02 15:13:06.821909 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:06.821874    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:13:07.845109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:07.845076    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:13:10.982221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:10.982187    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:12.105429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:12.105356    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"]
Nov 02 15:13:12.123043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:12.122944    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79317dd-x8cjp"]
Nov 02 15:13:13.969824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:13.968923    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7965717-df85-4506-a64a-a873f1825972" path="/var/lib/kubelet/pods/f7965717-df85-4506-a64a-a873f1825972/volumes"
Nov 02 15:13:17.034829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:17.034791    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"]
Nov 02 15:13:17.044158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:17.044126    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d970ba-tnbvd"]
Nov 02 15:13:17.891815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:17.891775    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f760fc72-2e8b-484a-955a-35a029acc55b" path="/var/lib/kubelet/pods/f760fc72-2e8b-484a-955a-35a029acc55b/volumes"
Nov 02 15:13:24.174673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:24.174641    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"]
Nov 02 15:13:24.217050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:24.217024    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"]
Nov 02 15:13:25.093691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:25.093651    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"]
Nov 02 15:13:25.119856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:25.119749    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"]
Nov 02 15:13:26.034098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:26.034064    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"]
Nov 02 15:13:26.043471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:26.043443    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3140768-nd8dm"]
Nov 02 15:13:27.912398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:27.908291    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8" path="/var/lib/kubelet/pods/ccff4436-d16c-4f4a-ab8c-b7f473c1a5b8/volumes"
Nov 02 15:13:29.180100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:29.180053    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" podUID="d0d0b8ed-a61e-4176-899c-356a60c31c16" containerName="payload-processing" containerID="cri-o://26e9a85bb9f15f382de1b75fe747445c0aa012497801264b08a0603fbcb7efbb" gracePeriod=25
Nov 02 15:13:29.224961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:29.224375    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5" podUID="badf915b-c1e2-46b9-8c41-80bef554df20" containerName="payload-pre-processing" containerID="cri-o://7f8c7f68133d7161e4b0bb5a9e7b5cb4c5aed5af512605fa83be992025f8d4b2" gracePeriod=25
Nov 02 15:13:29.428329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:29.426627    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:29.813390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:29.791744    5041 generic.go:356] "Generic (PLEG): container finished" podID="badf915b-c1e2-46b9-8c41-80bef554df20" containerID="7f8c7f68133d7161e4b0bb5a9e7b5cb4c5aed5af512605fa83be992025f8d4b2" exitCode=0
Nov 02 15:13:29.813390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:29.792097    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5" event={"ID":"badf915b-c1e2-46b9-8c41-80bef554df20","Type":"ContainerDied","Data":"7f8c7f68133d7161e4b0bb5a9e7b5cb4c5aed5af512605fa83be992025f8d4b2"}
Nov 02 15:13:29.877437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:29.860465    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.095453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.079049    5041 generic.go:356] "Generic (PLEG): container finished" podID="d0d0b8ed-a61e-4176-899c-356a60c31c16" containerID="26e9a85bb9f15f382de1b75fe747445c0aa012497801264b08a0603fbcb7efbb" exitCode=0
Nov 02 15:13:30.128398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.101433    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" podUID="ddfbe668-8d57-415d-a41d-003c050128a8" containerName="payload-processing" containerID="cri-o://41efe1a79665d61956e52fc00867abc61518be7b2e8de85db705a15b410c9973" gracePeriod=25
Nov 02 15:13:30.183354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.183306    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" podUID="6ebdd3dc-d531-486f-9e99-178f94da21d3" containerName="payload-pre-processing" containerID="cri-o://b321eceb525b7eadaa5b3097a46381d1c0bfc650c8c820ac98ed3e9ad04546c3" gracePeriod=25
Nov 02 15:13:30.313496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.311674    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.412438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.313865    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.524382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.524323    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded"
Nov 02 15:13:30.532922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.532872    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Nov 02 15:13:30.696401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.668816    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.738416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.738057    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.905399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.862345    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.905399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.862565    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" event={"ID":"d0d0b8ed-a61e-4176-899c-356a60c31c16","Type":"ContainerDied","Data":"26e9a85bb9f15f382de1b75fe747445c0aa012497801264b08a0603fbcb7efbb"}
Nov 02 15:13:30.905399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.862673    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:13:30.905399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.862717    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:13:30.911796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:30.911763    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"]
Nov 02 15:13:31.502403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.474486    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.653442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.566612    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.702603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.566651    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.702603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.590494    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.733455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.637929    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:13:31.733455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.726561    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:13:31.733455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.638245    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.733455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.727461    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:13:31.752460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.742449    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.752460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.744172    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:13:31.782389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.778918    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6658ba6-sq77h"]
Nov 02 15:13:31.795284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.786897    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:13:31.838630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.833341    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.838630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:31.833634    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:13:32.066493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:32.043171    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:13:32.638095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:32.638030    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.836017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:32.790334    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.837208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:32.836568    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89" podUID="39e554bb-b67f-471a-8990-5fa031445730" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.847461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:32.839348    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": context deadline exceeded"
Nov 02 15:13:32.871442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:32.848130    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:13:33.074391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.073808    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg" podUID="214f8315-f258-40d4-bb11-bfae94d43ff1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.074391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.074230    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:33.074391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.074285    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.091498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.091461    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:33.093616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.093581    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.132435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.110650    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.132435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.126561    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" podUID="d0d0b8ed-a61e-4176-899c-356a60c31c16" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.158:50052\" within 1s: context deadline exceeded"
Nov 02 15:13:33.132435 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:33.129026    5041 logging.go:55] [core] [Channel #1988 SubChannel #1989]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.158:50052", ServerName: "10.132.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.158:50052: operation was canceled"
Nov 02 15:13:33.257236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.257187    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" podUID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.76:50052\" within 1s: context deadline exceeded"
Nov 02 15:13:33.298483 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:33.298446    5041 logging.go:55] [core] [Channel #1999 SubChannel #2012]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.76:50052", ServerName: "10.132.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.76:50052: operation was canceled"
Nov 02 15:13:33.467513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.431105    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Nov 02 15:13:33.758459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.699904    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.830388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:33.794520    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.113445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.110889    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.184660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.184610    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" podUID="e93e760f-6eec-46e3-824c-2708f9df02f4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded"
Nov 02 15:13:34.204533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.204486    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.217970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.197839    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.486576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.197864    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.486576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.252343    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.514399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.514224    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:34.514399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.514290    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.836824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.836729    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.897809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.852586    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.897809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.882476    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded"
Nov 02 15:13:34.897809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.884147    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063e963a-d3a8-438d-91a1-8dd7dd9892f1" path="/var/lib/kubelet/pods/063e963a-d3a8-438d-91a1-8dd7dd9892f1/volumes"
Nov 02 15:13:34.946408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.927550    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg" podUID="214f8315-f258-40d4-bb11-bfae94d43ff1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.969563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:34.969465    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kbk89" podUID="39e554bb-b67f-471a-8990-5fa031445730" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.014928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.014892    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume\") pod \"badf915b-c1e2-46b9-8c41-80bef554df20\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") "
Nov 02 15:13:35.015165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.015146    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4\" (UniqueName: \"kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4\") pod \"badf915b-c1e2-46b9-8c41-80bef554df20\" (UID: \"badf915b-c1e2-46b9-8c41-80bef554df20\") "
Nov 02 15:13:35.015466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.015447    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume\") pod \"d0d0b8ed-a61e-4176-899c-356a60c31c16\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") "
Nov 02 15:13:35.015704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.015686    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj\" (UniqueName: \"kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj\") pod \"d0d0b8ed-a61e-4176-899c-356a60c31c16\" (UID: \"d0d0b8ed-a61e-4176-899c-356a60c31c16\") "
Nov 02 15:13:35.017697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.017673    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume" pod "badf915b-c1e2-46b9-8c41-80bef554df20" (UID: "badf915b-c1e2-46b9-8c41-80bef554df20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:35.080437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.076389    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.103213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.103093    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.109767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.109730    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume" pod "d0d0b8ed-a61e-4176-899c-356a60c31c16" (UID: "d0d0b8ed-a61e-4176-899c-356a60c31c16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:35.162394 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:35.159132    5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.28s"
Nov 02 15:13:35.200055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.196115    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badf915b-c1e2-46b9-8c41-80bef554df20-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:35.200055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.196148    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d0b8ed-a61e-4176-899c-356a60c31c16-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:35.243211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.210483    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.295013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.293814    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj" pod "d0d0b8ed-a61e-4176-899c-356a60c31c16" (UID: "d0d0b8ed-a61e-4176-899c-356a60c31c16"). InnerVolumeSpecName "kube-api-access-4hztj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:35.295013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.293917    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4" pod "badf915b-c1e2-46b9-8c41-80bef554df20" (UID: "badf915b-c1e2-46b9-8c41-80bef554df20"). InnerVolumeSpecName "kube-api-access-b9wq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:35.308594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.308561    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hztj\" (UniqueName: \"kubernetes.io/projected/d0d0b8ed-a61e-4176-899c-356a60c31c16-kube-api-access-4hztj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:35.308776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.308757    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wq4\" (UniqueName: \"kubernetes.io/projected/badf915b-c1e2-46b9-8c41-80bef554df20-kube-api-access-b9wq4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:35.326780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.326746    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"
Nov 02 15:13:35.400218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.400145    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:13:35.400481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.400458    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5" event={"ID":"badf915b-c1e2-46b9-8c41-80bef554df20","Type":"ContainerDied","Data":"80714de60e2c54b1a0c7b2fafc90867f229b949405af122d9a24613f77a75bc1"}
Nov 02 15:13:35.400645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.400632    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:13:35.400758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.400748    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:13:35.400850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.400837    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv" event={"ID":"d0d0b8ed-a61e-4176-899c-356a60c31c16","Type":"ContainerDied","Data":"ffccdc16d41ea667e0de711af7b567c70f8a9a4c4ac47bbc3ad7e77c4165c211"}
Nov 02 15:13:35.401239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401204    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"]
Nov 02 15:13:35.401407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401394    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:13:35.401537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401526    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:13:35.401631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401621    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"]
Nov 02 15:13:35.401742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401732    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:13:35.401841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401833    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:13:35.401927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.401918    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6078c8d7-nn7lr"]
Nov 02 15:13:35.402036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402027    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:13:35.402133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402124    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:13:35.402217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402208    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"]
Nov 02 15:13:35.402316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402307    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"]
Nov 02 15:13:35.402419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402409    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc270cc-r4dj5"]
Nov 02 15:13:35.402515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402506    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"]
Nov 02 15:13:35.402615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.402606    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"]
Nov 02 15:13:35.403778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.403757    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d0d0b8ed-a61e-4176-899c-356a60c31c16" containerName="payload-processing"
Nov 02 15:13:35.408839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.408817    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d0b8ed-a61e-4176-899c-356a60c31c16" containerName="payload-processing"
Nov 02 15:13:35.411034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.411014    5041 cpu_manager.go:414] "containerMap: removing container" podUID="badf915b-c1e2-46b9-8c41-80bef554df20" containerName="payload-pre-processing"
Nov 02 15:13:35.437605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.437564    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf915b-c1e2-46b9-8c41-80bef554df20" containerName="payload-pre-processing"
Nov 02 15:13:35.438044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.438021    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d0b8ed-a61e-4176-899c-356a60c31c16" containerName="payload-processing"
Nov 02 15:13:35.438234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.438206    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf915b-c1e2-46b9-8c41-80bef554df20" containerName="payload-pre-processing"
Nov 02 15:13:35.452849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.452825    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"]
Nov 02 15:13:35.453237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.453218    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"]
Nov 02 15:13:35.453427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.453411    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245255d5-hccgd"]
Nov 02 15:13:35.496384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.477645    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" podUID="390dccfe-5b8e-4be8-a109-1beeead39593" containerName="maas-api" containerID="cri-o://cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b" gracePeriod=30
Nov 02 15:13:35.603951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.603919    5041 scope.go:122] "RemoveContainer" containerID="7f8c7f68133d7161e4b0bb5a9e7b5cb4c5aed5af512605fa83be992025f8d4b2"
Nov 02 15:13:35.604781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.604716    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.869836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.850700    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.912033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.874617    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:35.912033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.874682    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.958417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:35.957950    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" podUID="6ebdd3dc-d531-486f-9e99-178f94da21d3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Nov 02 15:13:35.972915 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:35.961610    5041 logging.go:55] [core] [Channel #2016 SubChannel #2017]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.160:50052: operation was canceled"
Nov 02 15:13:36.044399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.037889    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.170157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.085057    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw" podUID="f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" containerName="maas-api" containerID="cri-o://4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5" gracePeriod=30
Nov 02 15:13:36.170488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.122512    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.171499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.133222    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" podUID="8ba06348-016b-4350-9f77-671cde29fed6" containerName="istio-proxy" containerID="cri-o://2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981" gracePeriod=30
Nov 02 15:13:36.172633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.172604    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" podUID="ddfbe668-8d57-415d-a41d-003c050128a8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.159:50052\" within 1s: context deadline exceeded"
Nov 02 15:13:36.172907 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:36.172889    5041 logging.go:55] [core] [Channel #2022 SubChannel #2024]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.159:50052", ServerName: "10.132.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.159:50052: operation was canceled"
Nov 02 15:13:36.190071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.190025    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.237884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.193104    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:13:36.237884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.194404    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:13:36.237884 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:36.199756    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390dccfe_5b8e_4be8_a109_1beeead39593.slice/crio-conmon-cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d0b8ed_a61e_4176_899c_356a60c31c16.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390dccfe_5b8e_4be8_a109_1beeead39593.slice/crio-cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:13:36.237884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.236948    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:13:36.288944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.288894    5041 generic.go:356] "Generic (PLEG): container finished" podID="6ebdd3dc-d531-486f-9e99-178f94da21d3" containerID="b321eceb525b7eadaa5b3097a46381d1c0bfc650c8c820ac98ed3e9ad04546c3" exitCode=0
Nov 02 15:13:36.289234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.289200    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd4zf\" (UniqueName: \"kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.289943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.289727    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.289943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.289798    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.312274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.312181    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e" path="/var/lib/kubelet/pods/91cb1c5b-d0e6-45b0-a5a1-47ff4d397d8e/volumes"
Nov 02 15:13:36.313191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.313166    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebacc0eb-768f-426a-8c2a-fc6d12fa44ca" path="/var/lib/kubelet/pods/ebacc0eb-768f-426a-8c2a-fc6d12fa44ca/volumes"
Nov 02 15:13:36.314833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.314791    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcba7116-075b-4789-aecf-774639e6757c" path="/var/lib/kubelet/pods/fcba7116-075b-4789-aecf-774639e6757c/volumes"
Nov 02 15:13:36.315939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.315910    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:13:36.316106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.316090    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"]
Nov 02 15:13:36.316279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.316253    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" event={"ID":"6ebdd3dc-d531-486f-9e99-178f94da21d3","Type":"ContainerDied","Data":"b321eceb525b7eadaa5b3097a46381d1c0bfc650c8c820ac98ed3e9ad04546c3"}
Nov 02 15:13:36.316929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.316867    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"]
Nov 02 15:13:36.317149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.317126    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:13:36.318423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.318271    5041 generic.go:356] "Generic (PLEG): container finished" podID="ddfbe668-8d57-415d-a41d-003c050128a8" containerID="41efe1a79665d61956e52fc00867abc61518be7b2e8de85db705a15b410c9973" exitCode=0
Nov 02 15:13:36.329214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.322776    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" event={"ID":"ddfbe668-8d57-415d-a41d-003c050128a8","Type":"ContainerDied","Data":"41efe1a79665d61956e52fc00867abc61518be7b2e8de85db705a15b410c9973"}
Nov 02 15:13:36.332661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.332642    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"
Nov 02 15:13:36.333510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.333494    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:13:36.360314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.360282    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:13:36.397006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.394394    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr\" (UniqueName: \"kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr\") pod \"6ebdd3dc-d531-486f-9e99-178f94da21d3\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") "
Nov 02 15:13:36.397006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.394608    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume\") pod \"6ebdd3dc-d531-486f-9e99-178f94da21d3\" (UID: \"6ebdd3dc-d531-486f-9e99-178f94da21d3\") "
Nov 02 15:13:36.397006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.395077    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd4zf\" (UniqueName: \"kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.397006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.395925    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.397006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.396018    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.399009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.398978    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume" pod "6ebdd3dc-d531-486f-9e99-178f94da21d3" (UID: "6ebdd3dc-d531-486f-9e99-178f94da21d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:36.402169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.399187    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr" pod "6ebdd3dc-d531-486f-9e99-178f94da21d3" (UID: "6ebdd3dc-d531-486f-9e99-178f94da21d3"). InnerVolumeSpecName "kube-api-access-2jxmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:36.404914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.403261    5041 scope.go:122] "RemoveContainer" containerID="26e9a85bb9f15f382de1b75fe747445c0aa012497801264b08a0603fbcb7efbb"
Nov 02 15:13:36.404914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.403321    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.409886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.409855    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.457112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.457080    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd4zf\" (UniqueName: \"kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf\") pod \"maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.462062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.462039    5041 scope.go:122] "RemoveContainer" containerID="b321eceb525b7eadaa5b3097a46381d1c0bfc650c8c820ac98ed3e9ad04546c3"
Nov 02 15:13:36.498475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.497242    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume\") pod \"ddfbe668-8d57-415d-a41d-003c050128a8\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") "
Nov 02 15:13:36.498475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.497751    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85\" (UniqueName: \"kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85\") pod \"ddfbe668-8d57-415d-a41d-003c050128a8\" (UID: \"ddfbe668-8d57-415d-a41d-003c050128a8\") "
Nov 02 15:13:36.499233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.499159    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume" pod "ddfbe668-8d57-415d-a41d-003c050128a8" (UID: "ddfbe668-8d57-415d-a41d-003c050128a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:36.501228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.501205    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jxmr\" (UniqueName: \"kubernetes.io/projected/6ebdd3dc-d531-486f-9e99-178f94da21d3-kube-api-access-2jxmr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:36.501341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.501231    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfbe668-8d57-415d-a41d-003c050128a8-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:36.501341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.501247    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebdd3dc-d531-486f-9e99-178f94da21d3-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:36.504785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.504761    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85" pod "ddfbe668-8d57-415d-a41d-003c050128a8" (UID: "ddfbe668-8d57-415d-a41d-003c050128a8"). InnerVolumeSpecName "kube-api-access-tdv85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:36.574124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.574099    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:13:36.602961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.602930    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdv85\" (UniqueName: \"kubernetes.io/projected/ddfbe668-8d57-415d-a41d-003c050128a8-kube-api-access-tdv85\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:36.682723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.682669    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:36.756435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.755483    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs\") pod \"390dccfe-5b8e-4be8-a109-1beeead39593\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") "
Nov 02 15:13:36.756435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.755583    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\") pod \"390dccfe-5b8e-4be8-a109-1beeead39593\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") "
Nov 02 15:13:36.756435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.755783    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca\") pod \"390dccfe-5b8e-4be8-a109-1beeead39593\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") "
Nov 02 15:13:36.756435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.755969    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g\" (UniqueName: \"kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g\") pod \"390dccfe-5b8e-4be8-a109-1beeead39593\" (UID: \"390dccfe-5b8e-4be8-a109-1beeead39593\") "
Nov 02 15:13:36.785648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.785555    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca" pod "390dccfe-5b8e-4be8-a109-1beeead39593" (UID: "390dccfe-5b8e-4be8-a109-1beeead39593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:36.899485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.872245    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390dccfe-5b8e-4be8-a109-1beeead39593-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:36.959403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.943968    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls" pod "390dccfe-5b8e-4be8-a109-1beeead39593" (UID: "390dccfe-5b8e-4be8-a109-1beeead39593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:36.959403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.944536    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g" pod "390dccfe-5b8e-4be8-a109-1beeead39593" (UID: "390dccfe-5b8e-4be8-a109-1beeead39593"). InnerVolumeSpecName "kube-api-access-b8j4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:37.004393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:36.978260    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8j4g\" (UniqueName: \"kubernetes.io/projected/390dccfe-5b8e-4be8-a109-1beeead39593-kube-api-access-b8j4g\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.033013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.009523    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.079406 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:37.073144    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:13:37.118197 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.118157    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs" pod "390dccfe-5b8e-4be8-a109-1beeead39593" (UID: "390dccfe-5b8e-4be8-a109-1beeead39593"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:37.153543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.153439    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390dccfe-5b8e-4be8-a109-1beeead39593-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.184861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.184395    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:13:37.296203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.294877    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca\") pod \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") "
Nov 02 15:13:37.296203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.294942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr\" (UniqueName: \"kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr\") pod \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") "
Nov 02 15:13:37.296203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.295050    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs\") pod \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") "
Nov 02 15:13:37.296203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.295265    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca" pod "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" (UID: "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:37.296203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.295280    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls\") pod \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\" (UID: \"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a\") "
Nov 02 15:13:37.296203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.295906    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.298798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.298758    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls" pod "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" (UID: "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:37.303701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.302046    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr" pod "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" (UID: "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a"). InnerVolumeSpecName "kube-api-access-s4bsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:37.303701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.302174    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs" pod "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" (UID: "f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:37.368784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.368758    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"
Nov 02 15:13:37.368968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.368782    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs" event={"ID":"ddfbe668-8d57-415d-a41d-003c050128a8","Type":"ContainerDied","Data":"f11cf1388a02041804c25c16c997bd0b47c545e782fd4a07580119d24e2e8317"}
Nov 02 15:13:37.368968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.368841    5041 scope.go:122] "RemoveContainer" containerID="41efe1a79665d61956e52fc00867abc61518be7b2e8de85db705a15b410c9973"
Nov 02 15:13:37.373258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.373229    5041 generic.go:356] "Generic (PLEG): container finished" podID="390dccfe-5b8e-4be8-a109-1beeead39593" containerID="cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b" exitCode=0
Nov 02 15:13:37.373426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.373321    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"
Nov 02 15:13:37.373426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.373340    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" event={"ID":"390dccfe-5b8e-4be8-a109-1beeead39593","Type":"ContainerDied","Data":"cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b"}
Nov 02 15:13:37.373426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.373401    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp" event={"ID":"390dccfe-5b8e-4be8-a109-1beeead39593","Type":"ContainerDied","Data":"744869935d2b3abecc5b91906d889fe4c2e7ab7df9031a2846e7456f2e3e6969"}
Nov 02 15:13:37.376037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.376013    5041 generic.go:356] "Generic (PLEG): container finished" podID="f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" containerID="4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5" exitCode=0
Nov 02 15:13:37.376154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.376110    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw" event={"ID":"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a","Type":"ContainerDied","Data":"4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5"}
Nov 02 15:13:37.376154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.376137    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw" event={"ID":"f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a","Type":"ContainerDied","Data":"5d99b41a318a54e21591576be3068d280a9611f29e84f6ba031388e88be5b905"}
Nov 02 15:13:37.376344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.376186    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"
Nov 02 15:13:37.381235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.381214    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"
Nov 02 15:13:37.384382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.384341    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz" event={"ID":"6ebdd3dc-d531-486f-9e99-178f94da21d3","Type":"ContainerDied","Data":"037b139b78a805ccdf5fdf03f0e85ab90af1dd6cbd8e5806aa5a015cf90b05ec"}
Nov 02 15:13:37.399065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.399041    5041 scope.go:122] "RemoveContainer" containerID="cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b"
Nov 02 15:13:37.404613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.404587    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4bsr\" (UniqueName: \"kubernetes.io/projected/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-kube-api-access-s4bsr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.404713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.404615    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.404713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.404632    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:37.449988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.449960    5041 scope.go:122] "RemoveContainer" containerID="cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b"
Nov 02 15:13:37.451076 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:37.451048    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b\": container with ID starting with cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b not found: ID does not exist" containerID="cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b"
Nov 02 15:13:37.453435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.451082    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b"} err="failed to get container status \"cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b\": rpc error: code = NotFound desc = could not find container \"cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b\": container with ID starting with cd2d989993bb4fd5057dd0526e89d2b4302e06735db41f0b1eeae6de74ef438b not found: ID does not exist"
Nov 02 15:13:37.453553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.453441    5041 scope.go:122] "RemoveContainer" containerID="4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5"
Nov 02 15:13:37.478180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.478161    5041 scope.go:122] "RemoveContainer" containerID="4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5"
Nov 02 15:13:37.479048 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:37.479019    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5\": container with ID starting with 4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5 not found: ID does not exist" containerID="4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5"
Nov 02 15:13:37.479434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.479065    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5"} err="failed to get container status \"4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5\": rpc error: code = NotFound desc = could not find container \"4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5\": container with ID starting with 4ea9c57711326589edf3a7288b3c5eb71c60b13658fce8c603187f3a7351dfe5 not found: ID does not exist"
Nov 02 15:13:37.546668 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:37.546613    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0c6223_0ebf_4b10_a9df_d1441380bde1.slice/crio-c78a3ae6007f7b8323f13357d25b5b9ea88fe1afc0a20fab93a39c646bc75b09 WatchSource:0}: Error finding container c78a3ae6007f7b8323f13357d25b5b9ea88fe1afc0a20fab93a39c646bc75b09: Status 404 returned error can't find the container with id c78a3ae6007f7b8323f13357d25b5b9ea88fe1afc0a20fab93a39c646bc75b09
Nov 02 15:13:37.547130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.547114    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"]
Nov 02 15:13:37.554905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.554882    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0621b336-6f4dfdb47f-v5hzv"]
Nov 02 15:13:37.564850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.564825    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"]
Nov 02 15:13:37.679740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.679704    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"]
Nov 02 15:13:37.685519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.685484    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0621b336-b48f546b7-5dfz5"]
Nov 02 15:13:37.907526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.907492    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf915b-c1e2-46b9-8c41-80bef554df20" path="/var/lib/kubelet/pods/badf915b-c1e2-46b9-8c41-80bef554df20/volumes"
Nov 02 15:13:37.908615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.908589    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d0b8ed-a61e-4176-899c-356a60c31c16" path="/var/lib/kubelet/pods/d0d0b8ed-a61e-4176-899c-356a60c31c16/volumes"
Nov 02 15:13:37.909119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.909100    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"]
Nov 02 15:13:37.911591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.911569    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0621b336-7fc8cff64c-4lllp"]
Nov 02 15:13:37.947718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.947689    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"]
Nov 02 15:13:37.954148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.954117    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2fcc8a1-574db9cf45-rxwqs"]
Nov 02 15:13:37.966251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.966225    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"]
Nov 02 15:13:37.975449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:37.975428    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2fcc8a1-66f46877c5-2l2pz"]
Nov 02 15:13:38.063022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:38.062990    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"]
Nov 02 15:13:38.070499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:38.070466    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2fcc8a1-669d444c-z5fvw"]
Nov 02 15:13:38.401815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:38.401774    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5" event={"ID":"0e0c6223-0ebf-4b10-a9df-d1441380bde1","Type":"ContainerStarted","Data":"ab7eff34b57c665be055e7dd3e236316dcc30bffd49d9ad1c80ecfd77235b484"}
Nov 02 15:13:38.402085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:38.402054    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5" event={"ID":"0e0c6223-0ebf-4b10-a9df-d1441380bde1","Type":"ContainerStarted","Data":"c78a3ae6007f7b8323f13357d25b5b9ea88fe1afc0a20fab93a39c646bc75b09"}
Nov 02 15:13:39.026673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.026621    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5" podStartSLOduration=6.026606467 podStartE2EDuration="6.026606467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:38.459586427 +0000 UTC m=+1070.717655877" watchObservedRunningTime="2026-11-02 15:13:39.026606467 +0000 UTC m=+1071.284675920"
Nov 02 15:13:39.033514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.033481    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"]
Nov 02 15:13:39.045175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.045144    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effba133-6hhj8"]
Nov 02 15:13:39.490652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.490612    5041 generic.go:356] "Generic (PLEG): container finished" podID="0e0c6223-0ebf-4b10-a9df-d1441380bde1" containerID="ab7eff34b57c665be055e7dd3e236316dcc30bffd49d9ad1c80ecfd77235b484" exitCode=0
Nov 02 15:13:39.491008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.490983    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5" event={"ID":"0e0c6223-0ebf-4b10-a9df-d1441380bde1","Type":"ContainerDied","Data":"ab7eff34b57c665be055e7dd3e236316dcc30bffd49d9ad1c80ecfd77235b484"}
Nov 02 15:13:39.892763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.892720    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390dccfe-5b8e-4be8-a109-1beeead39593" path="/var/lib/kubelet/pods/390dccfe-5b8e-4be8-a109-1beeead39593/volumes"
Nov 02 15:13:39.894973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.893580    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebdd3dc-d531-486f-9e99-178f94da21d3" path="/var/lib/kubelet/pods/6ebdd3dc-d531-486f-9e99-178f94da21d3/volumes"
Nov 02 15:13:39.894973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.894242    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfbe668-8d57-415d-a41d-003c050128a8" path="/var/lib/kubelet/pods/ddfbe668-8d57-415d-a41d-003c050128a8/volumes"
Nov 02 15:13:39.895989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.895967    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df53e0e0-d860-4127-81ef-45a536f76ce3" path="/var/lib/kubelet/pods/df53e0e0-d860-4127-81ef-45a536f76ce3/volumes"
Nov 02 15:13:39.896841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:39.896821    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" path="/var/lib/kubelet/pods/f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a/volumes"
Nov 02 15:13:40.695329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.695303    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:40.735326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.735292    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf\" (UniqueName: \"kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf\") pod \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") "
Nov 02 15:13:40.735477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.735355    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca\") pod \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") "
Nov 02 15:13:40.735713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.735689    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca\") pod \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\" (UID: \"0e0c6223-0ebf-4b10-a9df-d1441380bde1\") "
Nov 02 15:13:40.736459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.736430    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca" pod "0e0c6223-0ebf-4b10-a9df-d1441380bde1" (UID: "0e0c6223-0ebf-4b10-a9df-d1441380bde1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:40.737769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.737741    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf" pod "0e0c6223-0ebf-4b10-a9df-d1441380bde1" (UID: "0e0c6223-0ebf-4b10-a9df-d1441380bde1"). InnerVolumeSpecName "kube-api-access-hd4zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:40.737863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.737834    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca" pod "0e0c6223-0ebf-4b10-a9df-d1441380bde1" (UID: "0e0c6223-0ebf-4b10-a9df-d1441380bde1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:40.744507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.744480    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd4zf\" (UniqueName: \"kubernetes.io/projected/0e0c6223-0ebf-4b10-a9df-d1441380bde1-kube-api-access-hd4zf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:40.744622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.744514    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:40.744622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:40.744530    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0c6223-0ebf-4b10-a9df-d1441380bde1-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:41.327394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.326499    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh" podUID="8d366a77-f87a-4111-8166-9fa955d2e681" containerName="payload-processing" containerID="cri-o://ec9d681f01d0b8617f9e9c6f14d0e3ee1690630a3344cb15cb3d41d238307443" gracePeriod=25
Nov 02 15:13:41.327394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.326443    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" podUID="e93e760f-6eec-46e3-824c-2708f9df02f4" containerName="payload-pre-processing" containerID="cri-o://b5055aeed3d9a4aeb3cc6a17277954e0614dddceb619d647958322db21b41c6a" gracePeriod=25
Nov 02 15:13:41.518350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.518313    5041 generic.go:356] "Generic (PLEG): container finished" podID="e93e760f-6eec-46e3-824c-2708f9df02f4" containerID="b5055aeed3d9a4aeb3cc6a17277954e0614dddceb619d647958322db21b41c6a" exitCode=0
Nov 02 15:13:41.518541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.518435    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" event={"ID":"e93e760f-6eec-46e3-824c-2708f9df02f4","Type":"ContainerDied","Data":"b5055aeed3d9a4aeb3cc6a17277954e0614dddceb619d647958322db21b41c6a"}
Nov 02 15:13:41.520468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.520443    5041 generic.go:356] "Generic (PLEG): container finished" podID="8d366a77-f87a-4111-8166-9fa955d2e681" containerID="ec9d681f01d0b8617f9e9c6f14d0e3ee1690630a3344cb15cb3d41d238307443" exitCode=0
Nov 02 15:13:41.520593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.520535    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh" event={"ID":"8d366a77-f87a-4111-8166-9fa955d2e681","Type":"ContainerDied","Data":"ec9d681f01d0b8617f9e9c6f14d0e3ee1690630a3344cb15cb3d41d238307443"}
Nov 02 15:13:41.522852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.522669    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5" event={"ID":"0e0c6223-0ebf-4b10-a9df-d1441380bde1","Type":"ContainerDied","Data":"c78a3ae6007f7b8323f13357d25b5b9ea88fe1afc0a20fab93a39c646bc75b09"}
Nov 02 15:13:41.522852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.522714    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78a3ae6007f7b8323f13357d25b5b9ea88fe1afc0a20fab93a39c646bc75b09"
Nov 02 15:13:41.522852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.522783    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"
Nov 02 15:13:41.706780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.706747    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:13:41.794412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.791085    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume\") pod \"e93e760f-6eec-46e3-824c-2708f9df02f4\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") "
Nov 02 15:13:41.794412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.791227    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg\" (UniqueName: \"kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg\") pod \"e93e760f-6eec-46e3-824c-2708f9df02f4\" (UID: \"e93e760f-6eec-46e3-824c-2708f9df02f4\") "
Nov 02 15:13:41.794412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.791593    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:13:41.794412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.792511    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume" pod "e93e760f-6eec-46e3-824c-2708f9df02f4" (UID: "e93e760f-6eec-46e3-824c-2708f9df02f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:41.794412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.793461    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93e760f-6eec-46e3-824c-2708f9df02f4-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:41.835599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.829349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg" pod "e93e760f-6eec-46e3-824c-2708f9df02f4" (UID: "e93e760f-6eec-46e3-824c-2708f9df02f4"). InnerVolumeSpecName "kube-api-access-d6frg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:41.953402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.953209    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9\" (UniqueName: \"kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9\") pod \"8d366a77-f87a-4111-8166-9fa955d2e681\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") "
Nov 02 15:13:41.953402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.953325    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume\") pod \"8d366a77-f87a-4111-8166-9fa955d2e681\" (UID: \"8d366a77-f87a-4111-8166-9fa955d2e681\") "
Nov 02 15:13:41.983421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.982444    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6frg\" (UniqueName: \"kubernetes.io/projected/e93e760f-6eec-46e3-824c-2708f9df02f4-kube-api-access-d6frg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:41.983421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.982968    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume" pod "8d366a77-f87a-4111-8166-9fa955d2e681" (UID: "8d366a77-f87a-4111-8166-9fa955d2e681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:42.006387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:41.985940    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9" pod "8d366a77-f87a-4111-8166-9fa955d2e681" (UID: "8d366a77-f87a-4111-8166-9fa955d2e681"). InnerVolumeSpecName "kube-api-access-f8xz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:42.084523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.084486    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8xz9\" (UniqueName: \"kubernetes.io/projected/8d366a77-f87a-4111-8166-9fa955d2e681-kube-api-access-f8xz9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:42.084748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.084730    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d366a77-f87a-4111-8166-9fa955d2e681-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:42.145226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.145190    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"]
Nov 02 15:13:42.171429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.171267    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-302bb933-gfxrh"]
Nov 02 15:13:42.536922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.536885    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2" event={"ID":"e93e760f-6eec-46e3-824c-2708f9df02f4","Type":"ContainerDied","Data":"e8f47d8c30369c4c28e2b4d35db969cb8d4f2f2192e81e955e86a20e7ed77e7a"}
Nov 02 15:13:42.536922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.536902    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"
Nov 02 15:13:42.537159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.536955    5041 scope.go:122] "RemoveContainer" containerID="b5055aeed3d9a4aeb3cc6a17277954e0614dddceb619d647958322db21b41c6a"
Nov 02 15:13:42.539110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.539078    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"
Nov 02 15:13:42.539236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.539111    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh" event={"ID":"8d366a77-f87a-4111-8166-9fa955d2e681","Type":"ContainerDied","Data":"2beaaa989dd3e2e2458e311b6acf9744b73e4c4cacc1cee406c08eb7d018c210"}
Nov 02 15:13:42.552762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.552732    5041 scope.go:122] "RemoveContainer" containerID="ec9d681f01d0b8617f9e9c6f14d0e3ee1690630a3344cb15cb3d41d238307443"
Nov 02 15:13:42.648757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.648710    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"]
Nov 02 15:13:42.654967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.654935    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-649ad0be-5b6b756d5c-fcqrh"]
Nov 02 15:13:42.689250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.689206    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"]
Nov 02 15:13:42.714059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.714026    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-649ad0be-dcdbbfb5f-fk9h2"]
Nov 02 15:13:42.720254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.720222    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"]
Nov 02 15:13:42.721009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.720990    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ddfbe668-8d57-415d-a41d-003c050128a8" containerName="payload-processing"
Nov 02 15:13:42.721009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721005    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfbe668-8d57-415d-a41d-003c050128a8" containerName="payload-processing"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721022    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" containerName="maas-api"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721028    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" containerName="maas-api"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721046    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0e0c6223-0ebf-4b10-a9df-d1441380bde1" containerName="revoke-keys"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721051    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0c6223-0ebf-4b10-a9df-d1441380bde1" containerName="revoke-keys"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721072    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d366a77-f87a-4111-8166-9fa955d2e681" containerName="payload-processing"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721077    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d366a77-f87a-4111-8166-9fa955d2e681" containerName="payload-processing"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721088    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e93e760f-6eec-46e3-824c-2708f9df02f4" containerName="payload-pre-processing"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721092    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93e760f-6eec-46e3-824c-2708f9df02f4" containerName="payload-pre-processing"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721105    5041 cpu_manager.go:414] "containerMap: removing container" podUID="390dccfe-5b8e-4be8-a109-1beeead39593" containerName="maas-api"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721109    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390dccfe-5b8e-4be8-a109-1beeead39593" containerName="maas-api"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721120    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6ebdd3dc-d531-486f-9e99-178f94da21d3" containerName="payload-pre-processing"
Nov 02 15:13:42.721164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721124    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebdd3dc-d531-486f-9e99-178f94da21d3" containerName="payload-pre-processing"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721233    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0c6223-0ebf-4b10-a9df-d1441380bde1" containerName="revoke-keys"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721243    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d366a77-f87a-4111-8166-9fa955d2e681" containerName="payload-processing"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721250    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93e760f-6eec-46e3-824c-2708f9df02f4" containerName="payload-pre-processing"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721257    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="390dccfe-5b8e-4be8-a109-1beeead39593" containerName="maas-api"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721264    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebdd3dc-d531-486f-9e99-178f94da21d3" containerName="payload-pre-processing"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721272    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfbe668-8d57-415d-a41d-003c050128a8" containerName="payload-processing"
Nov 02 15:13:42.721811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.721277    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ac236d-1f99-47ae-b01b-7bd53d7d5e2a" containerName="maas-api"
Nov 02 15:13:42.725720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.725700    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:42.749634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.749607    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-991657be\""
Nov 02 15:13:42.749785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.749770    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-991657be-dockercfg-qcfks\""
Nov 02 15:13:42.760941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.760916    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"]
Nov 02 15:13:42.881434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.870410    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24bk\" (UniqueName: \"kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk\") pod \"payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:42.918607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.870657    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:42.981412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.962976    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"]
Nov 02 15:13:42.981412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:42.968503    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.056428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.026966    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume\") pod \"payload-processing-e2e-ait-991657be-b79d4c49f-2l42j\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") " pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.056428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.027262    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24bk\" (UniqueName: \"kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk\") pod \"payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:43.056428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.027410    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt7jr\" (UniqueName: \"kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr\") pod \"payload-processing-e2e-ait-991657be-b79d4c49f-2l42j\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") " pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.056428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.027504    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:43.075000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.061650    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:43.114385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.105298    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24bk\" (UniqueName: \"kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk\") pod \"payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:43.137689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.134674    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt7jr\" (UniqueName: \"kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr\") pod \"payload-processing-e2e-ait-991657be-b79d4c49f-2l42j\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") " pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.137689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.135249    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume\") pod \"payload-processing-e2e-ait-991657be-b79d4c49f-2l42j\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") " pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.137689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.136293    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume\") pod \"payload-processing-e2e-ait-991657be-b79d4c49f-2l42j\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") " pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.185407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.174397    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"]
Nov 02 15:13:43.193417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.191938    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt7jr\" (UniqueName: \"kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr\") pod \"payload-processing-e2e-ait-991657be-b79d4c49f-2l42j\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") " pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.360384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.357717    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:43.416464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.413626    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:43.867155 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:43.861556    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4087f33d_f5e2_4150_ae4b_3ca5252d3162.slice/crio-f882ebb30458f4e6bc5954f8caecd6884f88c1631a22d019693db0976ada2a0b WatchSource:0}: Error finding container f882ebb30458f4e6bc5954f8caecd6884f88c1631a22d019693db0976ada2a0b: Status 404 returned error can't find the container with id f882ebb30458f4e6bc5954f8caecd6884f88c1631a22d019693db0976ada2a0b
Nov 02 15:13:43.867155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.862169    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"]
Nov 02 15:13:43.880641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.874648    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"]
Nov 02 15:13:43.904300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.904269    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf40389-00ec-4183-a747-b4e6b0426b59" path="/var/lib/kubelet/pods/1cf40389-00ec-4183-a747-b4e6b0426b59/volumes"
Nov 02 15:13:43.904861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.904834    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d366a77-f87a-4111-8166-9fa955d2e681" path="/var/lib/kubelet/pods/8d366a77-f87a-4111-8166-9fa955d2e681/volumes"
Nov 02 15:13:43.905308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.905286    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93e760f-6eec-46e3-824c-2708f9df02f4" path="/var/lib/kubelet/pods/e93e760f-6eec-46e3-824c-2708f9df02f4/volumes"
Nov 02 15:13:43.973674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.973646    5041 patch_prober.go:29] interesting pod/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": read tcp 10.132.0.2:57534->10.132.0.156:15021: read: connection reset by peer" start-of-body=
Nov 02 15:13:43.973786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:43.973696    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" podUID="8ba06348-016b-4350-9f77-671cde29fed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": read tcp 10.132.0.2:57534->10.132.0.156:15021: read: connection reset by peer"
Nov 02 15:13:44.089707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.089674    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"]
Nov 02 15:13:44.127438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.127314    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"]
Nov 02 15:13:44.258168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.258141    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.426678    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.426773    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.426873    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.427010    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.427110    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.427300    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.427336    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.427395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.427448    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy\") pod \"8ba06348-016b-4350-9f77-671cde29fed6\" (UID: \"8ba06348-016b-4350-9f77-671cde29fed6\") "
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.428624    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:44.431144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.428820    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:44.450258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.441541    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:13:44.450258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.442292    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "kube-api-access-5wjpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:44.450258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.442682    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:44.450258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.442974    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:44.450258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.446474    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:44.525645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.525598    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:44.525958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.525931    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy" pod "8ba06348-016b-4350-9f77-671cde29fed6" (UID: "8ba06348-016b-4350-9f77-671cde29fed6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:44.529310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529284    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529311    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba06348-016b-4350-9f77-671cde29fed6-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529321    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529330    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba06348-016b-4350-9f77-671cde29fed6-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529338    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529346    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529357    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjpc\" (UniqueName: \"kubernetes.io/projected/8ba06348-016b-4350-9f77-671cde29fed6-kube-api-access-5wjpc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529384    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.529493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.529406    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba06348-016b-4350-9f77-671cde29fed6-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:44.573736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.573703    5041 generic.go:356] "Generic (PLEG): container finished" podID="8ba06348-016b-4350-9f77-671cde29fed6" containerID="2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981" exitCode=0
Nov 02 15:13:44.573852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.573759    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"
Nov 02 15:13:44.573852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.573783    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" event={"ID":"8ba06348-016b-4350-9f77-671cde29fed6","Type":"ContainerDied","Data":"2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981"}
Nov 02 15:13:44.573985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.573888    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j" event={"ID":"8ba06348-016b-4350-9f77-671cde29fed6","Type":"ContainerDied","Data":"a9b160d5934cd34e1a18c4fa36cdb7a26450c3f72059afeb8c2046636efe220d"}
Nov 02 15:13:44.573985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.573922    5041 scope.go:122] "RemoveContainer" containerID="2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981"
Nov 02 15:13:44.575942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.575909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j" event={"ID":"4087f33d-f5e2-4150-ae4b-3ca5252d3162","Type":"ContainerStarted","Data":"fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30"}
Nov 02 15:13:44.576039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.575962    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j" event={"ID":"4087f33d-f5e2-4150-ae4b-3ca5252d3162","Type":"ContainerStarted","Data":"f882ebb30458f4e6bc5954f8caecd6884f88c1631a22d019693db0976ada2a0b"}
Nov 02 15:13:44.576039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.575993    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:44.577932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.577909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k" event={"ID":"888646d2-32a2-4eb5-8ed1-63112f041266","Type":"ContainerStarted","Data":"26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390"}
Nov 02 15:13:44.578030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.577952    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k" event={"ID":"888646d2-32a2-4eb5-8ed1-63112f041266","Type":"ContainerStarted","Data":"2631179cc1f05a75b3ffbcdc4d0104f4b9798cb5ec187d38708a428853f38d62"}
Nov 02 15:13:44.578030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.577998    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:44.587759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.587740    5041 scope.go:122] "RemoveContainer" containerID="2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981"
Nov 02 15:13:44.588033 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:44.588013    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981\": container with ID starting with 2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981 not found: ID does not exist" containerID="2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981"
Nov 02 15:13:44.588081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.588041    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981"} err="failed to get container status \"2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981\": rpc error: code = NotFound desc = could not find container \"2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981\": container with ID starting with 2511fe3b05db9d31dd0641881c5953065c7abeb25ce9654d4b95b0f34a16b981 not found: ID does not exist"
Nov 02 15:13:44.625288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.625240    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k" podStartSLOduration=2.625227963 podStartE2EDuration="2.625227963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:44.621667245 +0000 UTC m=+1076.879736695" watchObservedRunningTime="2026-11-02 15:13:44.625227963 +0000 UTC m=+1076.883297415"
Nov 02 15:13:44.658247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.658221    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"]
Nov 02 15:13:44.667127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.667102    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c2fcc8a1-openshift-default-6978d84757-q4r4j"]
Nov 02 15:13:44.696303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:44.696234    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j" podStartSLOduration=2.696224021 podStartE2EDuration="2.696224021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:44.690513092 +0000 UTC m=+1076.948582558" watchObservedRunningTime="2026-11-02 15:13:44.696224021 +0000 UTC m=+1076.954293470"
Nov 02 15:13:45.185706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.185676    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"]
Nov 02 15:13:45.186127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.185867    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7" podUID="079839b5-efef-4707-bab1-15156d3a2275" containerName="maas-api" containerID="cri-o://c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46" gracePeriod=30
Nov 02 15:13:45.601766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.601737    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:13:45.606802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.606771    5041 generic.go:356] "Generic (PLEG): container finished" podID="079839b5-efef-4707-bab1-15156d3a2275" containerID="c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46" exitCode=0
Nov 02 15:13:45.607814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.607791    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"
Nov 02 15:13:45.607962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.607929    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7" event={"ID":"079839b5-efef-4707-bab1-15156d3a2275","Type":"ContainerDied","Data":"c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46"}
Nov 02 15:13:45.608035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.607973    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7" event={"ID":"079839b5-efef-4707-bab1-15156d3a2275","Type":"ContainerDied","Data":"359e18c1419f35a88ae5657b35fac5c68072f045273595dd8f65f2dd5e2545b1"}
Nov 02 15:13:45.608102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.608087    5041 scope.go:122] "RemoveContainer" containerID="c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46"
Nov 02 15:13:45.628103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.627781    5041 scope.go:122] "RemoveContainer" containerID="c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46"
Nov 02 15:13:45.628640 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:45.628538    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46\": container with ID starting with c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46 not found: ID does not exist" containerID="c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46"
Nov 02 15:13:45.628640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.628576    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46"} err="failed to get container status \"c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46\": rpc error: code = NotFound desc = could not find container \"c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46\": container with ID starting with c052d1b3a2305e7c5b3ff9b2bf76030033b8981eed5e70eabedbaa18c71bea46 not found: ID does not exist"
Nov 02 15:13:45.705592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.704126    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls\") pod \"079839b5-efef-4707-bab1-15156d3a2275\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") "
Nov 02 15:13:45.705592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.704233    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca\") pod \"079839b5-efef-4707-bab1-15156d3a2275\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") "
Nov 02 15:13:45.705592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.704326    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr\" (UniqueName: \"kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr\") pod \"079839b5-efef-4707-bab1-15156d3a2275\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") "
Nov 02 15:13:45.705592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.704383    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs\") pod \"079839b5-efef-4707-bab1-15156d3a2275\" (UID: \"079839b5-efef-4707-bab1-15156d3a2275\") "
Nov 02 15:13:45.714313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.708329    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca" pod "079839b5-efef-4707-bab1-15156d3a2275" (UID: "079839b5-efef-4707-bab1-15156d3a2275"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:45.818433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.812497    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs" pod "079839b5-efef-4707-bab1-15156d3a2275" (UID: "079839b5-efef-4707-bab1-15156d3a2275"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:45.878393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.875200    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls" pod "079839b5-efef-4707-bab1-15156d3a2275" (UID: "079839b5-efef-4707-bab1-15156d3a2275"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:45.949423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.898583    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr" pod "079839b5-efef-4707-bab1-15156d3a2275" (UID: "079839b5-efef-4707-bab1-15156d3a2275"). InnerVolumeSpecName "kube-api-access-mz2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:45.966104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.965916    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:45.966104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.965950    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079839b5-efef-4707-bab1-15156d3a2275-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:45.966104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.965965    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz2vr\" (UniqueName: \"kubernetes.io/projected/079839b5-efef-4707-bab1-15156d3a2275-kube-api-access-mz2vr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:45.966104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:45.965979    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079839b5-efef-4707-bab1-15156d3a2275-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.109836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:46.109806    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba06348-016b-4350-9f77-671cde29fed6" path="/var/lib/kubelet/pods/8ba06348-016b-4350-9f77-671cde29fed6/volumes"
Nov 02 15:13:46.110627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:46.110527    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"]
Nov 02 15:13:46.116648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:46.116619    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57202685-4xxkb"]
Nov 02 15:13:46.139170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:46.139125    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"]
Nov 02 15:13:46.151804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:46.151769    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-649ad0be-7b7477b75b-9whj7"]
Nov 02 15:13:47.056268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.056235    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"]
Nov 02 15:13:47.068790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.068760    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56368a0-ghbv7"]
Nov 02 15:13:47.075007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.074983    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"]
Nov 02 15:13:47.080523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.080500    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed65845-449b2"]
Nov 02 15:13:47.876022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.875990    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"]
Nov 02 15:13:47.876329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.876300    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx" podUID="075a0fbc-afa6-47f3-932f-93de6743307e" containerName="maas-api" containerID="cri-o://24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c" gracePeriod=30
Nov 02 15:13:47.896035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.896000    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079839b5-efef-4707-bab1-15156d3a2275" path="/var/lib/kubelet/pods/079839b5-efef-4707-bab1-15156d3a2275/volumes"
Nov 02 15:13:47.896555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.896533    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f332039-b9c4-4dfe-8f76-ce03ef383c52" path="/var/lib/kubelet/pods/1f332039-b9c4-4dfe-8f76-ce03ef383c52/volumes"
Nov 02 15:13:47.897215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.897195    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bff10ba-80e4-4be1-873a-53759ead5447" path="/var/lib/kubelet/pods/2bff10ba-80e4-4be1-873a-53759ead5447/volumes"
Nov 02 15:13:47.898218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:47.898157    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e4c139-3746-4611-a926-e1ec04ac43a4" path="/var/lib/kubelet/pods/55e4c139-3746-4611-a926-e1ec04ac43a4/volumes"
Nov 02 15:13:48.162831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.162759    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"]
Nov 02 15:13:48.163611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.163588    5041 cpu_manager.go:414] "containerMap: removing container" podUID="079839b5-efef-4707-bab1-15156d3a2275" containerName="maas-api"
Nov 02 15:13:48.163611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.163607    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079839b5-efef-4707-bab1-15156d3a2275" containerName="maas-api"
Nov 02 15:13:48.163783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.163648    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8ba06348-016b-4350-9f77-671cde29fed6" containerName="istio-proxy"
Nov 02 15:13:48.163783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.163654    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba06348-016b-4350-9f77-671cde29fed6" containerName="istio-proxy"
Nov 02 15:13:48.163783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.163777    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba06348-016b-4350-9f77-671cde29fed6" containerName="istio-proxy"
Nov 02 15:13:48.163887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.163792    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="079839b5-efef-4707-bab1-15156d3a2275" containerName="maas-api"
Nov 02 15:13:48.167266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.167247    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.203297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.203268    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8a3fb0-openshift-default-dockercfg-wpbsj\""
Nov 02 15:13:48.208157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.208127    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:48.220952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.220925    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"]
Nov 02 15:13:48.227468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227433    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227526    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227564    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227618    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227684    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjxr\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227747    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227881    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.227953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227937    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.228008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.227983    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.328744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.328702    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm\" (UniqueName: \"kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm\") pod \"075a0fbc-afa6-47f3-932f-93de6743307e\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") "
Nov 02 15:13:48.328942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.328821    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca\") pod \"075a0fbc-afa6-47f3-932f-93de6743307e\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") "
Nov 02 15:13:48.329058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.329012    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls\") pod \"075a0fbc-afa6-47f3-932f-93de6743307e\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") "
Nov 02 15:13:48.329221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.329202    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs\") pod \"075a0fbc-afa6-47f3-932f-93de6743307e\" (UID: \"075a0fbc-afa6-47f3-932f-93de6743307e\") "
Nov 02 15:13:48.329650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.329623    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.329976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.329954    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330031    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330052    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330113    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330402    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330687    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330717    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.330790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330772    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca" pod "075a0fbc-afa6-47f3-932f-93de6743307e" (UID: "075a0fbc-afa6-47f3-932f-93de6743307e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:48.330970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.330904    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.331571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.331543    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.331935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.331909    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.332069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.332039    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs" pod "075a0fbc-afa6-47f3-932f-93de6743307e" (UID: "075a0fbc-afa6-47f3-932f-93de6743307e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:48.332293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.332269    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.332431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.332411    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.332848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.332823    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjxr\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.333323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.333300    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:48.333461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.333329    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075a0fbc-afa6-47f3-932f-93de6743307e-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:48.333565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.333540    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm" pod "075a0fbc-afa6-47f3-932f-93de6743307e" (UID: "075a0fbc-afa6-47f3-932f-93de6743307e"). InnerVolumeSpecName "kube-api-access-s7fsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:48.333708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.333684    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls" pod "075a0fbc-afa6-47f3-932f-93de6743307e" (UID: "075a0fbc-afa6-47f3-932f-93de6743307e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:48.336637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.335659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.336637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.335844    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.354097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.354075    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjxr\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.354189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.354132    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token\") pod \"e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") " pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.440785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.439389    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075a0fbc-afa6-47f3-932f-93de6743307e-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:48.440785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.439427    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fsm\" (UniqueName: \"kubernetes.io/projected/075a0fbc-afa6-47f3-932f-93de6743307e-kube-api-access-s7fsm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:48.537793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.537529    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:48.821544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.820523    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"]
Nov 02 15:13:48.828577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.828544    5041 generic.go:356] "Generic (PLEG): container finished" podID="075a0fbc-afa6-47f3-932f-93de6743307e" containerID="24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c" exitCode=0
Nov 02 15:13:48.828929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.828905    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx" event={"ID":"075a0fbc-afa6-47f3-932f-93de6743307e","Type":"ContainerDied","Data":"24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c"}
Nov 02 15:13:48.829223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.829199    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx" event={"ID":"075a0fbc-afa6-47f3-932f-93de6743307e","Type":"ContainerDied","Data":"1361df79225063103d6df9bd25573b777d02a14643027db246d9212c4231bd66"}
Nov 02 15:13:48.829315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.829249    5041 scope.go:122] "RemoveContainer" containerID="24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c"
Nov 02 15:13:48.829411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.829140    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"
Nov 02 15:13:48.835222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.832747    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3bd1bf1-ddvl5"]
Nov 02 15:13:48.871330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.871275    5041 scope.go:122] "RemoveContainer" containerID="24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c"
Nov 02 15:13:48.873690 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:48.873660    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c\": container with ID starting with 24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c not found: ID does not exist" containerID="24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c"
Nov 02 15:13:48.873872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.873844    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c"} err="failed to get container status \"24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c\": rpc error: code = NotFound desc = could not find container \"24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c\": container with ID starting with 24f9d4ba4a4daaba6341aac39f7ed7397436b1c8de05e95f1b1b9aa4435a3b6c not found: ID does not exist"
Nov 02 15:13:48.999461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:48.998162    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"]
Nov 02 15:13:49.057500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.057466    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3bd1bf1-5dcf8d8896-pwpnx"]
Nov 02 15:13:49.090159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.090100    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn" podUID="bdcc045a-4708-4c3e-9b55-f3181a7dfed2" containerName="payload-processing" containerID="cri-o://d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33" gracePeriod=25
Nov 02 15:13:49.120243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.120215    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"]
Nov 02 15:13:49.128159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.128128    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx" podUID="53742354-8c0c-46bc-84e2-ab9bde2c321d" containerName="payload-pre-processing" containerID="cri-o://5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e" gracePeriod=25
Nov 02 15:13:49.138225 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:49.138202    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46272a17_1a46_4999_95d2_b25e6e57ddc0.slice/crio-9d3cef5b0dbc77d558e5fb3fd7758aa3e49213171d12d4fbb5785e8814e709e6 WatchSource:0}: Error finding container 9d3cef5b0dbc77d558e5fb3fd7758aa3e49213171d12d4fbb5785e8814e709e6: Status 404 returned error can't find the container with id 9d3cef5b0dbc77d558e5fb3fd7758aa3e49213171d12d4fbb5785e8814e709e6
Nov 02 15:13:49.234335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.234293    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:13:49.234711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.234450    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:13:49.234711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.234525    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:13:49.432862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.432510    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:13:49.459519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.459481    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:13:49.498198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.498134    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume\") pod \"53742354-8c0c-46bc-84e2-ab9bde2c321d\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") "
Nov 02 15:13:49.498433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.498414    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8\" (UniqueName: \"kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8\") pod \"53742354-8c0c-46bc-84e2-ab9bde2c321d\" (UID: \"53742354-8c0c-46bc-84e2-ab9bde2c321d\") "
Nov 02 15:13:49.498830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.498552    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume" pod "53742354-8c0c-46bc-84e2-ab9bde2c321d" (UID: "53742354-8c0c-46bc-84e2-ab9bde2c321d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:49.499093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.499075    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53742354-8c0c-46bc-84e2-ab9bde2c321d-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:49.500959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.500926    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8" pod "53742354-8c0c-46bc-84e2-ab9bde2c321d" (UID: "53742354-8c0c-46bc-84e2-ab9bde2c321d"). InnerVolumeSpecName "kube-api-access-m7qd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:49.600717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.600632    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f\" (UniqueName: \"kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f\") pod \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") "
Nov 02 15:13:49.600717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.600689    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume\") pod \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\" (UID: \"bdcc045a-4708-4c3e-9b55-f3181a7dfed2\") "
Nov 02 15:13:49.601098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.601071    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume" pod "bdcc045a-4708-4c3e-9b55-f3181a7dfed2" (UID: "bdcc045a-4708-4c3e-9b55-f3181a7dfed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:49.601335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.601313    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7qd8\" (UniqueName: \"kubernetes.io/projected/53742354-8c0c-46bc-84e2-ab9bde2c321d-kube-api-access-m7qd8\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:49.601466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.601340    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:49.602998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.602975    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f" pod "bdcc045a-4708-4c3e-9b55-f3181a7dfed2" (UID: "bdcc045a-4708-4c3e-9b55-f3181a7dfed2"). InnerVolumeSpecName "kube-api-access-jj46f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:49.702966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.702935    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj46f\" (UniqueName: \"kubernetes.io/projected/bdcc045a-4708-4c3e-9b55-f3181a7dfed2-kube-api-access-jj46f\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:13:49.841522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.841488    5041 generic.go:356] "Generic (PLEG): container finished" podID="bdcc045a-4708-4c3e-9b55-f3181a7dfed2" containerID="d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33" exitCode=0
Nov 02 15:13:49.841718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.841528    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"
Nov 02 15:13:49.841718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.841565    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn" event={"ID":"bdcc045a-4708-4c3e-9b55-f3181a7dfed2","Type":"ContainerDied","Data":"d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33"}
Nov 02 15:13:49.841718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.841625    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn" event={"ID":"bdcc045a-4708-4c3e-9b55-f3181a7dfed2","Type":"ContainerDied","Data":"94c972154a8c01faf4f5114f2e0bccee6e3e0dda15f6326ad48a9080f6164425"}
Nov 02 15:13:49.841718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.841656    5041 scope.go:122] "RemoveContainer" containerID="d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33"
Nov 02 15:13:49.844816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.844793    5041 generic.go:356] "Generic (PLEG): container finished" podID="53742354-8c0c-46bc-84e2-ab9bde2c321d" containerID="5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e" exitCode=0
Nov 02 15:13:49.844954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.844852    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"
Nov 02 15:13:49.844954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.844868    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx" event={"ID":"53742354-8c0c-46bc-84e2-ab9bde2c321d","Type":"ContainerDied","Data":"5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e"}
Nov 02 15:13:49.844954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.844903    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx" event={"ID":"53742354-8c0c-46bc-84e2-ab9bde2c321d","Type":"ContainerDied","Data":"5b34b7e2535fa20cd41014223072b1ede6419a93220bbc07d2e371f9c6454ff6"}
Nov 02 15:13:49.846610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.846583    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" event={"ID":"46272a17-1a46-4999-95d2-b25e6e57ddc0","Type":"ContainerStarted","Data":"57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"}
Nov 02 15:13:49.846731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.846630    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" event={"ID":"46272a17-1a46-4999-95d2-b25e6e57ddc0","Type":"ContainerStarted","Data":"9d3cef5b0dbc77d558e5fb3fd7758aa3e49213171d12d4fbb5785e8814e709e6"}
Nov 02 15:13:49.857408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.857336    5041 scope.go:122] "RemoveContainer" containerID="d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33"
Nov 02 15:13:49.857616 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:49.857592    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33\": container with ID starting with d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33 not found: ID does not exist" containerID="d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33"
Nov 02 15:13:49.857689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.857626    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33"} err="failed to get container status \"d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33\": rpc error: code = NotFound desc = could not find container \"d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33\": container with ID starting with d80c5f5c0f0fd1a768d39bd1dbcafc416b9c50c0dbe8a5a41f5fd25f8a46ce33 not found: ID does not exist"
Nov 02 15:13:49.857689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.857650    5041 scope.go:122] "RemoveContainer" containerID="5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e"
Nov 02 15:13:49.868554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.868539    5041 scope.go:122] "RemoveContainer" containerID="5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e"
Nov 02 15:13:49.868769 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:13:49.868746    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e\": container with ID starting with 5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e not found: ID does not exist" containerID="5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e"
Nov 02 15:13:49.868834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.868778    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e"} err="failed to get container status \"5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e\": rpc error: code = NotFound desc = could not find container \"5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e\": container with ID starting with 5fab76c502cca69869ecdb1f6e9357848478de060d885909f33f26945a3d960e not found: ID does not exist"
Nov 02 15:13:49.889927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.889903    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075a0fbc-afa6-47f3-932f-93de6743307e" path="/var/lib/kubelet/pods/075a0fbc-afa6-47f3-932f-93de6743307e/volumes"
Nov 02 15:13:49.890468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.890448    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0c6223-0ebf-4b10-a9df-d1441380bde1" path="/var/lib/kubelet/pods/0e0c6223-0ebf-4b10-a9df-d1441380bde1/volumes"
Nov 02 15:13:49.891000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.890978    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"]
Nov 02 15:13:49.896340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.896318    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3bd1bf1-7d65b995b5-jr6qn"]
Nov 02 15:13:49.943328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.943288    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podStartSLOduration=1.9432766799999999 podStartE2EDuration="1.94327668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:49.940068852 +0000 UTC m=+1082.198138309" watchObservedRunningTime="2026-11-02 15:13:49.94327668 +0000 UTC m=+1082.201346203"
Nov 02 15:13:49.982908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.982876    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"]
Nov 02 15:13:49.987983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:49.987957    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3bd1bf1-8bc75b6c-lvhmx"]
Nov 02 15:13:50.548777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:50.548739    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:50.573675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:50.573637    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:50.860501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:50.860407    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:50.861315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:50.861293    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:13:51.888921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:51.888889    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53742354-8c0c-46bc-84e2-ab9bde2c321d" path="/var/lib/kubelet/pods/53742354-8c0c-46bc-84e2-ab9bde2c321d/volumes"
Nov 02 15:13:51.889300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:51.889291    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcc045a-4708-4c3e-9b55-f3181a7dfed2" path="/var/lib/kubelet/pods/bdcc045a-4708-4c3e-9b55-f3181a7dfed2/volumes"
Nov 02 15:13:54.040613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.040578    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"]
Nov 02 15:13:54.048286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.048254    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bcb8-mr7cm"]
Nov 02 15:13:54.245752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.245718    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"]
Nov 02 15:13:54.246553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246529    5041 cpu_manager.go:414] "containerMap: removing container" podUID="075a0fbc-afa6-47f3-932f-93de6743307e" containerName="maas-api"
Nov 02 15:13:54.246553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246551    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075a0fbc-afa6-47f3-932f-93de6743307e" containerName="maas-api"
Nov 02 15:13:54.246745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246597    5041 cpu_manager.go:414] "containerMap: removing container" podUID="53742354-8c0c-46bc-84e2-ab9bde2c321d" containerName="payload-pre-processing"
Nov 02 15:13:54.246745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246607    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53742354-8c0c-46bc-84e2-ab9bde2c321d" containerName="payload-pre-processing"
Nov 02 15:13:54.246745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246661    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bdcc045a-4708-4c3e-9b55-f3181a7dfed2" containerName="payload-processing"
Nov 02 15:13:54.246745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246670    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcc045a-4708-4c3e-9b55-f3181a7dfed2" containerName="payload-processing"
Nov 02 15:13:54.246947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246860    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcc045a-4708-4c3e-9b55-f3181a7dfed2" containerName="payload-processing"
Nov 02 15:13:54.246947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246879    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="075a0fbc-afa6-47f3-932f-93de6743307e" containerName="maas-api"
Nov 02 15:13:54.246947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.246895    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="53742354-8c0c-46bc-84e2-ab9bde2c321d" containerName="payload-pre-processing"
Nov 02 15:13:54.251229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.251207    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.264649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.264627    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8a3fb0\""
Nov 02 15:13:54.265030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.265009    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8a3fb0-dockercfg-gx6rr\""
Nov 02 15:13:54.273435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.273403    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"]
Nov 02 15:13:54.630411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.490771    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.630411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.554067    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58j6\" (UniqueName: \"kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6\") pod \"payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.643190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.632908    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"]
Nov 02 15:13:54.676221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.674273    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.676221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.674414    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58j6\" (UniqueName: \"kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6\") pod \"payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.676221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.675988    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"]
Nov 02 15:13:54.676221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.676117    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:54.692809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.688427    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.740331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.739232    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58j6\" (UniqueName: \"kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6\") pod \"payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.775780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.775748    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbv2\" (UniqueName: \"kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2\") pod \"payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:54.775983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.775959    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:54.864953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.864919    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:54.920438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.889168    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbv2\" (UniqueName: \"kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2\") pod \"payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:54.920897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.920859    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:54.929727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.922237    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:54.930240 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:54.930211    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbv2\" (UniqueName: \"kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2\") pod \"payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:55.008334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.006643    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:55.047420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.047141    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"]
Nov 02 15:13:55.054287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.054257    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"]
Nov 02 15:13:55.061353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.061325    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d0ec2a-dqpnj"]
Nov 02 15:13:55.067561 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.067532    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8f6ec2-nr45f"]
Nov 02 15:13:55.354644 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:55.354613    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5b4e77_a767_424d_9626_b2e877f1ae7b.slice/crio-e76292ef6c3ec8ddb38509f7ec61f0af722d4a8d575374e71c851120167c9f63 WatchSource:0}: Error finding container e76292ef6c3ec8ddb38509f7ec61f0af722d4a8d575374e71c851120167c9f63: Status 404 returned error can't find the container with id e76292ef6c3ec8ddb38509f7ec61f0af722d4a8d575374e71c851120167c9f63
Nov 02 15:13:55.355771 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.355745    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"]
Nov 02 15:13:55.398115 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:55.398091    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5704f35f_3ee9_4a3a_96b9_9d659d474ee7.slice/crio-919e0f06c9c4b7edff268b86c8285fb018156597ad4883344ecfbf1315418b64 WatchSource:0}: Error finding container 919e0f06c9c4b7edff268b86c8285fb018156597ad4883344ecfbf1315418b64: Status 404 returned error can't find the container with id 919e0f06c9c4b7edff268b86c8285fb018156597ad4883344ecfbf1315418b64
Nov 02 15:13:55.399967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.399946    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"]
Nov 02 15:13:55.610407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.610287    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:13:55.611187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.611163    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:13:55.889538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.889470    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712fd78e-59cc-4719-bc0b-534045981ec2" path="/var/lib/kubelet/pods/712fd78e-59cc-4719-bc0b-534045981ec2/volumes"
Nov 02 15:13:55.889997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.889982    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbed397-26d7-40c8-9b41-b3a5e47da1a2" path="/var/lib/kubelet/pods/bdbed397-26d7-40c8-9b41-b3a5e47da1a2/volumes"
Nov 02 15:13:55.890490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.890462    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec55a09e-8298-4e6a-b3c5-20f7e777e2be" path="/var/lib/kubelet/pods/ec55a09e-8298-4e6a-b3c5-20f7e777e2be/volumes"
Nov 02 15:13:55.980221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.980187    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" event={"ID":"5704f35f-3ee9-4a3a-96b9-9d659d474ee7","Type":"ContainerStarted","Data":"fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c"}
Nov 02 15:13:55.980356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.980237    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" event={"ID":"5704f35f-3ee9-4a3a-96b9-9d659d474ee7","Type":"ContainerStarted","Data":"919e0f06c9c4b7edff268b86c8285fb018156597ad4883344ecfbf1315418b64"}
Nov 02 15:13:55.980356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.980259    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:13:55.982117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.982085    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" event={"ID":"5c5b4e77-a767-424d-9626-b2e877f1ae7b","Type":"ContainerStarted","Data":"ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a"}
Nov 02 15:13:55.982232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.982128    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" event={"ID":"5c5b4e77-a767-424d-9626-b2e877f1ae7b","Type":"ContainerStarted","Data":"e76292ef6c3ec8ddb38509f7ec61f0af722d4a8d575374e71c851120167c9f63"}
Nov 02 15:13:55.982232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:55.982170    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:13:56.018341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:56.018293    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" podStartSLOduration=2.01828023 podStartE2EDuration="2.01828023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:56.015556806 +0000 UTC m=+1088.273626255" watchObservedRunningTime="2026-11-02 15:13:56.01828023 +0000 UTC m=+1088.276349682"
Nov 02 15:13:56.052691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:56.052655    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" podStartSLOduration=2.052644022 podStartE2EDuration="2.052644022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:56.050663876 +0000 UTC m=+1088.308733347" watchObservedRunningTime="2026-11-02 15:13:56.052644022 +0000 UTC m=+1088.310713507"
Nov 02 15:13:58.228583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.228546    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"]
Nov 02 15:13:58.233481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.233461    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.248224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.248198    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-991657be-serving-cert\""
Nov 02 15:13:58.267813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.267785    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"]
Nov 02 15:13:58.302930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.302891    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.303098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.303074    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.303165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.303123    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.303278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.303253    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.406659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.406575    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.408003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.407971    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.433330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.433206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.433940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.433913    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.489535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.489439    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.509457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.490238    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.512135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.512091    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.515197 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.515174    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk\") pod \"maas-api-e2e-ait-991657be-5667d6d7f7-jblvl\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.553032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.552991    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:13:58.889078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:58.889051    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"]
Nov 02 15:13:58.890979 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:13:58.890958    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfef5470_e65d_4bc7_8332_38c069a37c1c.slice/crio-dc18ded47cc44a57e163a43a713f7fc7adac2d5c415ea09b111dc6d35d604f8e WatchSource:0}: Error finding container dc18ded47cc44a57e163a43a713f7fc7adac2d5c415ea09b111dc6d35d604f8e: Status 404 returned error can't find the container with id dc18ded47cc44a57e163a43a713f7fc7adac2d5c415ea09b111dc6d35d604f8e
Nov 02 15:13:59.089631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:13:59.089587    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl" event={"ID":"bfef5470-e65d-4bc7-8332-38c069a37c1c","Type":"ContainerStarted","Data":"dc18ded47cc44a57e163a43a713f7fc7adac2d5c415ea09b111dc6d35d604f8e"}
Nov 02 15:14:00.102785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:00.102754    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl" event={"ID":"bfef5470-e65d-4bc7-8332-38c069a37c1c","Type":"ContainerStarted","Data":"9174af792951a1e5a2fbcd34f5fb5fb894a6b89151733682a15966a17889c470"}
Nov 02 15:14:00.103292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:00.102813    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:14:00.188389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:00.184548    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl" podStartSLOduration=1.81000268 podStartE2EDuration="2.184522847s" totalImagesPullingTime="374.520167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:00.163722993 +0000 UTC m=+1092.421792450" watchObservedRunningTime="2026-11-02 15:14:00.184522847 +0000 UTC m=+1092.442592298"
Nov 02 15:14:01.038639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:01.038602    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"]
Nov 02 15:14:01.047454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:01.047425    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d60855-x5z2z"]
Nov 02 15:14:01.925985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:01.925950    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a78d457-513d-4016-a908-878b6d0c9f54" path="/var/lib/kubelet/pods/9a78d457-513d-4016-a908-878b6d0c9f54/volumes"
Nov 02 15:14:06.119658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:06.119617    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:14:06.930392 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:14:06.921548    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:14:07.022160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:07.021540    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:14:07.025392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:07.022956    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:14:08.086110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.086073    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"]
Nov 02 15:14:08.092562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.092535    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b89b69b-r6l6r"]
Nov 02 15:14:08.739242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.739204    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"]
Nov 02 15:14:08.743100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.743082    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.764248 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.764217    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-8a3fb0-serving-cert\""
Nov 02 15:14:08.767543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.767519    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"]
Nov 02 15:14:08.827290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.827258    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.827599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.827548    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.827793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.827772    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.827930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.827914    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz566\" (UniqueName: \"kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.929239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.929209    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.929435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.929284    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.929435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.929342    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.929582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.929469    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz566\" (UniqueName: \"kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.930453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.930431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.932003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.931982    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.932060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.931982    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:08.951052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:08.951030    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz566\" (UniqueName: \"kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566\") pod \"maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:09.059025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:09.058927    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:09.440720 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:09.440691    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b2d51e_6655_487e_8f08_0e76ac12e88c.slice/crio-2f899cc9603bcf81b910fc76e6f9d750cec229af16ef1588c07bfaeaf3c67b9a WatchSource:0}: Error finding container 2f899cc9603bcf81b910fc76e6f9d750cec229af16ef1588c07bfaeaf3c67b9a: Status 404 returned error can't find the container with id 2f899cc9603bcf81b910fc76e6f9d750cec229af16ef1588c07bfaeaf3c67b9a
Nov 02 15:14:09.442845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:09.442818    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"]
Nov 02 15:14:09.819030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:09.818957    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr" event={"ID":"23b2d51e-6655-487e-8f08-0e76ac12e88c","Type":"ContainerStarted","Data":"2f899cc9603bcf81b910fc76e6f9d750cec229af16ef1588c07bfaeaf3c67b9a"}
Nov 02 15:14:09.899952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:09.899922    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862edf1c-a264-42ed-9746-f7b2261e3dee" path="/var/lib/kubelet/pods/862edf1c-a264-42ed-9746-f7b2261e3dee/volumes"
Nov 02 15:14:10.859862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:10.859810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr" event={"ID":"23b2d51e-6655-487e-8f08-0e76ac12e88c","Type":"ContainerStarted","Data":"d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f"}
Nov 02 15:14:10.861057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:10.861017    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:10.913996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:10.912152    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr" podStartSLOduration=2.528225404 podStartE2EDuration="2.912136605s" totalImagesPullingTime="383.911201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:10.91093808 +0000 UTC m=+1103.169007530" watchObservedRunningTime="2026-11-02 15:14:10.912136605 +0000 UTC m=+1103.170206055"
Nov 02 15:14:11.035207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:11.035174    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"]
Nov 02 15:14:11.041141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:11.041101    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b88472-pfqm4"]
Nov 02 15:14:11.889426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:11.889394    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8761c5e7-df26-4f7b-9526-675d8c9995e3" path="/var/lib/kubelet/pods/8761c5e7-df26-4f7b-9526-675d8c9995e3/volumes"
Nov 02 15:14:17.892732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:17.892697    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:14:24.368791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.368755    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"]
Nov 02 15:14:24.373437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.373415    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.395945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.395918    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"]
Nov 02 15:14:24.445903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.445864    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:14:24.446065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.446033    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:14:24.509270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.509242    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.509446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.509285    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.509511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.509470    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9jx\" (UniqueName: \"kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.610511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.610481    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.610634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.610527    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.610684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.610643    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9jx\" (UniqueName: \"kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.611290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.611268    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.613048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.613020    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.633376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.633312    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9jx\" (UniqueName: \"kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx\") pod \"maas-api-revoke-sub-7f69f34a-mqd8w\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.689343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.689320    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:24.997437 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:24.997401    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286f1174_4d09_45d2_afbe_96e3c0003ce8.slice/crio-0d205009b7488b69fc89d153fbb37e207dcf975d8197eba30f62d8ee2af6f750 WatchSource:0}: Error finding container 0d205009b7488b69fc89d153fbb37e207dcf975d8197eba30f62d8ee2af6f750: Status 404 returned error can't find the container with id 0d205009b7488b69fc89d153fbb37e207dcf975d8197eba30f62d8ee2af6f750
Nov 02 15:14:24.998725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:24.998698    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"]
Nov 02 15:14:25.021275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:25.021239    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w" event={"ID":"286f1174-4d09-45d2-afbe-96e3c0003ce8","Type":"ContainerStarted","Data":"0d205009b7488b69fc89d153fbb37e207dcf975d8197eba30f62d8ee2af6f750"}
Nov 02 15:14:26.032837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:26.032794    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w" event={"ID":"286f1174-4d09-45d2-afbe-96e3c0003ce8","Type":"ContainerStarted","Data":"35b82db8f048c559e56e2645e7d90302680458b52e529b4fd2a284fa1943d9d1"}
Nov 02 15:14:26.069892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:26.069846    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w" podStartSLOduration=2.069829291 podStartE2EDuration="2.069829291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:26.068185209 +0000 UTC m=+1118.326254659" watchObservedRunningTime="2026-11-02 15:14:26.069829291 +0000 UTC m=+1118.327898743"
Nov 02 15:14:27.045948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:27.045911    5041 generic.go:356] "Generic (PLEG): container finished" podID="286f1174-4d09-45d2-afbe-96e3c0003ce8" containerID="35b82db8f048c559e56e2645e7d90302680458b52e529b4fd2a284fa1943d9d1" exitCode=0
Nov 02 15:14:27.046321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:27.045993    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w" event={"ID":"286f1174-4d09-45d2-afbe-96e3c0003ce8","Type":"ContainerDied","Data":"35b82db8f048c559e56e2645e7d90302680458b52e529b4fd2a284fa1943d9d1"}
Nov 02 15:14:28.226265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.226239    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:28.307278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.307204    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx\" (UniqueName: \"kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx\") pod \"286f1174-4d09-45d2-afbe-96e3c0003ce8\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") "
Nov 02 15:14:28.307432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.307401    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca\") pod \"286f1174-4d09-45d2-afbe-96e3c0003ce8\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") "
Nov 02 15:14:28.307500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.307474    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca\") pod \"286f1174-4d09-45d2-afbe-96e3c0003ce8\" (UID: \"286f1174-4d09-45d2-afbe-96e3c0003ce8\") "
Nov 02 15:14:28.307868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.307833    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca" pod "286f1174-4d09-45d2-afbe-96e3c0003ce8" (UID: "286f1174-4d09-45d2-afbe-96e3c0003ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:14:28.308405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.308378    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:28.309736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.309706    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx" pod "286f1174-4d09-45d2-afbe-96e3c0003ce8" (UID: "286f1174-4d09-45d2-afbe-96e3c0003ce8"). InnerVolumeSpecName "kube-api-access-nm9jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:14:28.310073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.310041    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca" pod "286f1174-4d09-45d2-afbe-96e3c0003ce8" (UID: "286f1174-4d09-45d2-afbe-96e3c0003ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:14:28.409886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.409857    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286f1174-4d09-45d2-afbe-96e3c0003ce8-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:28.409886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:28.409884    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9jx\" (UniqueName: \"kubernetes.io/projected/286f1174-4d09-45d2-afbe-96e3c0003ce8-kube-api-access-nm9jx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:29.068439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:29.068409    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"
Nov 02 15:14:29.068616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:29.068408    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w" event={"ID":"286f1174-4d09-45d2-afbe-96e3c0003ce8","Type":"ContainerDied","Data":"0d205009b7488b69fc89d153fbb37e207dcf975d8197eba30f62d8ee2af6f750"}
Nov 02 15:14:29.068616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:29.068536    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d205009b7488b69fc89d153fbb37e207dcf975d8197eba30f62d8ee2af6f750"
Nov 02 15:14:30.574452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.574416    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t7m7n-qffpv"]
Nov 02 15:14:30.575234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.575219    5041 cpu_manager.go:414] "containerMap: removing container" podUID="286f1174-4d09-45d2-afbe-96e3c0003ce8" containerName="revoke-keys"
Nov 02 15:14:30.575283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.575235    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286f1174-4d09-45d2-afbe-96e3c0003ce8" containerName="revoke-keys"
Nov 02 15:14:30.575357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.575348    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="286f1174-4d09-45d2-afbe-96e3c0003ce8" containerName="revoke-keys"
Nov 02 15:14:30.578536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.578518    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.600479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.600449    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t7m7n-qffpv"]
Nov 02 15:14:30.602350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.602330    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-czhhz\""
Nov 02 15:14:30.682638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.682606    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.682823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.682668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.682823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.682780    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvwz\" (UniqueName: \"kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.683003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.682987    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.784470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.784431    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvwz\" (UniqueName: \"kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.784709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.784693    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.784780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.784765    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.784844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.784830    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.785048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.785028    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.785584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.785560    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.785654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.785629    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.805333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.805303    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvwz\" (UniqueName: \"kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz\") pod \"periodic-gathering-t7m7n-qffpv\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") " pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:30.892342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:30.892278    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:14:31.206915 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:31.206887    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe12de84_1c2d_492b_88ac_c3ca2c327b03.slice/crio-61c7e3a7fd75dac459eda247b8a83bfd215258a79271fe955f6e34d4329dd81a WatchSource:0}: Error finding container 61c7e3a7fd75dac459eda247b8a83bfd215258a79271fe955f6e34d4329dd81a: Status 404 returned error can't find the container with id 61c7e3a7fd75dac459eda247b8a83bfd215258a79271fe955f6e34d4329dd81a
Nov 02 15:14:31.208782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:31.208748    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t7m7n-qffpv"]
Nov 02 15:14:32.175616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.175579    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7m7n-qffpv" event={"ID":"fe12de84-1c2d-492b-88ac-c3ca2c327b03","Type":"ContainerStarted","Data":"527153f63042b9fc6cfe45c5b051ac4e201ea4352d658f0302838950349102a8"}
Nov 02 15:14:32.176002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.175632    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7m7n-qffpv" event={"ID":"fe12de84-1c2d-492b-88ac-c3ca2c327b03","Type":"ContainerStarted","Data":"61c7e3a7fd75dac459eda247b8a83bfd215258a79271fe955f6e34d4329dd81a"}
Nov 02 15:14:32.176268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.176248    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"]
Nov 02 15:14:32.190738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.190712    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.204169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.204128    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:14:32.204286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.204213    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:14:32.206253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.206229    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"]
Nov 02 15:14:32.214616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.214575    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t7m7n-qffpv" podStartSLOduration=2.214564192 podStartE2EDuration="2.214564192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:32.211238521 +0000 UTC m=+1124.469307997" watchObservedRunningTime="2026-11-02 15:14:32.214564192 +0000 UTC m=+1124.472633648"
Nov 02 15:14:32.322813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.322782    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ps6\" (UniqueName: \"kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.322961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.322930    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.323168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.323144    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.424471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.424442    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.424639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.424624    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ps6\" (UniqueName: \"kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.424725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.424694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.425098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.425077    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.427072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.427015    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.445288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.445263    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ps6\" (UniqueName: \"kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6\") pod \"maas-api-revoke-keys-e2e-ait-991657be-4ff4p\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.508970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.508946    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:32.873613 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:32.873584    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode903add9_16cc_46f1_af47_f2f7088bddaa.slice/crio-44b48e3f35bd34372f953cac727f317071a34d2f133c0c084674e2936c8b62f8 WatchSource:0}: Error finding container 44b48e3f35bd34372f953cac727f317071a34d2f133c0c084674e2936c8b62f8: Status 404 returned error can't find the container with id 44b48e3f35bd34372f953cac727f317071a34d2f133c0c084674e2936c8b62f8
Nov 02 15:14:32.875522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:32.875493    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"]
Nov 02 15:14:33.305097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:33.304093    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p" event={"ID":"e903add9-16cc-46f1-af47-f2f7088bddaa","Type":"ContainerStarted","Data":"44b48e3f35bd34372f953cac727f317071a34d2f133c0c084674e2936c8b62f8"}
Nov 02 15:14:34.289398 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:14:34.286769    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode903add9_16cc_46f1_af47_f2f7088bddaa.slice/crio-8707500bd71d81707ad382b75c80902ed4458d1d041dff3569c0d48dadd9d684.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode903add9_16cc_46f1_af47_f2f7088bddaa.slice/crio-conmon-8707500bd71d81707ad382b75c80902ed4458d1d041dff3569c0d48dadd9d684.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:14:34.302447 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:14:34.294460    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode903add9_16cc_46f1_af47_f2f7088bddaa.slice/crio-8707500bd71d81707ad382b75c80902ed4458d1d041dff3569c0d48dadd9d684.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:14:34.434390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:34.427375    5041 generic.go:356] "Generic (PLEG): container finished" podID="e903add9-16cc-46f1-af47-f2f7088bddaa" containerID="8707500bd71d81707ad382b75c80902ed4458d1d041dff3569c0d48dadd9d684" exitCode=0
Nov 02 15:14:34.434390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:34.427563    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p" event={"ID":"e903add9-16cc-46f1-af47-f2f7088bddaa","Type":"ContainerDied","Data":"8707500bd71d81707ad382b75c80902ed4458d1d041dff3569c0d48dadd9d684"}
Nov 02 15:14:35.682181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.682150    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:35.771139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.771044    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6\" (UniqueName: \"kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6\") pod \"e903add9-16cc-46f1-af47-f2f7088bddaa\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") "
Nov 02 15:14:35.771314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.771165    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca\") pod \"e903add9-16cc-46f1-af47-f2f7088bddaa\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") "
Nov 02 15:14:35.771314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.771278    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca\") pod \"e903add9-16cc-46f1-af47-f2f7088bddaa\" (UID: \"e903add9-16cc-46f1-af47-f2f7088bddaa\") "
Nov 02 15:14:35.771572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.771534    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca" pod "e903add9-16cc-46f1-af47-f2f7088bddaa" (UID: "e903add9-16cc-46f1-af47-f2f7088bddaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:14:35.771979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.771954    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:35.773352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.773321    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca" pod "e903add9-16cc-46f1-af47-f2f7088bddaa" (UID: "e903add9-16cc-46f1-af47-f2f7088bddaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:14:35.773514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.773444    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6" pod "e903add9-16cc-46f1-af47-f2f7088bddaa" (UID: "e903add9-16cc-46f1-af47-f2f7088bddaa"). InnerVolumeSpecName "kube-api-access-r5ps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:14:35.873426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.873351    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ps6\" (UniqueName: \"kubernetes.io/projected/e903add9-16cc-46f1-af47-f2f7088bddaa-kube-api-access-r5ps6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:35.873426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:35.873397    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903add9-16cc-46f1-af47-f2f7088bddaa-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:36.043328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:36.043241    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"]
Nov 02 15:14:36.049639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:36.049603    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35c1533-ccs6s"]
Nov 02 15:14:36.470876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:36.470849    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"
Nov 02 15:14:36.471071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:36.470890    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p" event={"ID":"e903add9-16cc-46f1-af47-f2f7088bddaa","Type":"ContainerDied","Data":"44b48e3f35bd34372f953cac727f317071a34d2f133c0c084674e2936c8b62f8"}
Nov 02 15:14:36.471071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:36.470945    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b48e3f35bd34372f953cac727f317071a34d2f133c0c084674e2936c8b62f8"
Nov 02 15:14:36.769839 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:14:36.769759    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:14:37.891382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:37.891339    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70e113c-efe2-42fa-8473-ffa9667fe84e" path="/var/lib/kubelet/pods/f70e113c-efe2-42fa-8473-ffa9667fe84e/volumes"
Nov 02 15:14:42.696509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:42.696469    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"]
Nov 02 15:14:42.728955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:42.728914    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"]
Nov 02 15:14:45.650437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:45.650398    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" event={"ID":"46272a17-1a46-4999-95d2-b25e6e57ddc0","Type":"ContainerDied","Data":"57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"}
Nov 02 15:14:45.650987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:45.650401    5041 generic.go:356] "Generic (PLEG): container finished" podID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerID="57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080" exitCode=0
Nov 02 15:14:45.651059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:45.651012    5041 scope.go:122] "RemoveContainer" containerID="57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"
Nov 02 15:14:45.652620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:45.652592    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:45.652684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:45.652662    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:45.652726 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:45.652694    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:46.663091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:46.663058    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" event={"ID":"46272a17-1a46-4999-95d2-b25e6e57ddc0","Type":"ContainerStarted","Data":"67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb"}
Nov 02 15:14:47.538929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:47.538893    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:14:47.539802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:47.539769    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 15:14:47.539950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:47.539858    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 15:14:47.697345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:47.697303    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j" podUID="4087f33d-f5e2-4150-ae4b-3ca5252d3162" containerName="payload-processing" containerID="cri-o://fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30" gracePeriod=25
Nov 02 15:14:47.732394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:47.729428    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k" podUID="888646d2-32a2-4eb5-8ed1-63112f041266" containerName="payload-pre-processing" containerID="cri-o://26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390" gracePeriod=25
Nov 02 15:14:48.145455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.145417    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:14:48.161616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.161582    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume\") pod \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") "
Nov 02 15:14:48.161911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.161887    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr\" (UniqueName: \"kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr\") pod \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\" (UID: \"4087f33d-f5e2-4150-ae4b-3ca5252d3162\") "
Nov 02 15:14:48.163711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.163685    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume" pod "4087f33d-f5e2-4150-ae4b-3ca5252d3162" (UID: "4087f33d-f5e2-4150-ae4b-3ca5252d3162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:14:48.165283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.165258    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr" pod "4087f33d-f5e2-4150-ae4b-3ca5252d3162" (UID: "4087f33d-f5e2-4150-ae4b-3ca5252d3162"). InnerVolumeSpecName "kube-api-access-lt7jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:14:48.257076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.256482    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:14:48.264160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.264133    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4087f33d-f5e2-4150-ae4b-3ca5252d3162-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:48.264292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.264271    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt7jr\" (UniqueName: \"kubernetes.io/projected/4087f33d-f5e2-4150-ae4b-3ca5252d3162-kube-api-access-lt7jr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:48.365294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.365194    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk\" (UniqueName: \"kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk\") pod \"888646d2-32a2-4eb5-8ed1-63112f041266\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") "
Nov 02 15:14:48.365533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.365346    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume\") pod \"888646d2-32a2-4eb5-8ed1-63112f041266\" (UID: \"888646d2-32a2-4eb5-8ed1-63112f041266\") "
Nov 02 15:14:48.365824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.365790    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume" pod "888646d2-32a2-4eb5-8ed1-63112f041266" (UID: "888646d2-32a2-4eb5-8ed1-63112f041266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:14:48.366241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.366212    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888646d2-32a2-4eb5-8ed1-63112f041266-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:48.367353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.367328    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk" pod "888646d2-32a2-4eb5-8ed1-63112f041266" (UID: "888646d2-32a2-4eb5-8ed1-63112f041266"). InnerVolumeSpecName "kube-api-access-x24bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:14:48.503387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.503331    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24bk\" (UniqueName: \"kubernetes.io/projected/888646d2-32a2-4eb5-8ed1-63112f041266-kube-api-access-x24bk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:48.544387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.543433    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 15:14:48.544387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.543511    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 15:14:48.757388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.756238    5041 generic.go:356] "Generic (PLEG): container finished" podID="888646d2-32a2-4eb5-8ed1-63112f041266" containerID="26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390" exitCode=0
Nov 02 15:14:48.757388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.756590    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k" event={"ID":"888646d2-32a2-4eb5-8ed1-63112f041266","Type":"ContainerDied","Data":"26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390"}
Nov 02 15:14:48.757388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.756745    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k" event={"ID":"888646d2-32a2-4eb5-8ed1-63112f041266","Type":"ContainerDied","Data":"2631179cc1f05a75b3ffbcdc4d0104f4b9798cb5ec187d38708a428853f38d62"}
Nov 02 15:14:48.757388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.756794    5041 scope.go:122] "RemoveContainer" containerID="26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390"
Nov 02 15:14:48.757388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.757077    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"
Nov 02 15:14:48.778851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.776107    5041 generic.go:356] "Generic (PLEG): container finished" podID="4087f33d-f5e2-4150-ae4b-3ca5252d3162" containerID="fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30" exitCode=0
Nov 02 15:14:48.778851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.776304    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j" event={"ID":"4087f33d-f5e2-4150-ae4b-3ca5252d3162","Type":"ContainerDied","Data":"fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30"}
Nov 02 15:14:48.778851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.776354    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j" event={"ID":"4087f33d-f5e2-4150-ae4b-3ca5252d3162","Type":"ContainerDied","Data":"f882ebb30458f4e6bc5954f8caecd6884f88c1631a22d019693db0976ada2a0b"}
Nov 02 15:14:48.778851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.776535    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"
Nov 02 15:14:48.802638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.801907    5041 scope.go:122] "RemoveContainer" containerID="26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390"
Nov 02 15:14:48.803149 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:14:48.803117    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390\": container with ID starting with 26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390 not found: ID does not exist" containerID="26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390"
Nov 02 15:14:48.803280 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.803157    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390"} err="failed to get container status \"26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390\": rpc error: code = NotFound desc = could not find container \"26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390\": container with ID starting with 26d30ca87911d7e328ec7c64bebdb4ef590ec75920e3fa815cf2788b4cde5390 not found: ID does not exist"
Nov 02 15:14:48.803280 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.803183    5041 scope.go:122] "RemoveContainer" containerID="fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30"
Nov 02 15:14:48.862212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.862184    5041 scope.go:122] "RemoveContainer" containerID="fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30"
Nov 02 15:14:48.862687 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:14:48.862661    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30\": container with ID starting with fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30 not found: ID does not exist" containerID="fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30"
Nov 02 15:14:48.862784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.862696    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30"} err="failed to get container status \"fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30\": rpc error: code = NotFound desc = could not find container \"fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30\": container with ID starting with fb8f1c047236d8bf6a29ed9cedfbe9ec032efd4609bc35132bfa37914c137e30 not found: ID does not exist"
Nov 02 15:14:48.891242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.891214    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"]
Nov 02 15:14:48.899819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.899789    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-991657be-5f6568cd99-vqd4k"]
Nov 02 15:14:48.943830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.943794    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"]
Nov 02 15:14:48.949567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:48.949537    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-991657be-b79d4c49f-2l42j"]
Nov 02 15:14:49.039978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.039894    5041 scope.go:122] "RemoveContainer" containerID="5e04164fc651633337570809bf8e7b4e263b843e19e2ff7041dcf886b8858afe"
Nov 02 15:14:49.052058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.052036    5041 scope.go:122] "RemoveContainer" containerID="e19f2bc5dfc26b62ef2a38a25b7c6178917cdbbc62fd9cd68ac69e986670c792"
Nov 02 15:14:49.068358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.065542    5041 scope.go:122] "RemoveContainer" containerID="7d03f81f9da98a546be10e3d02407528e3c72b71f502a6a9d35d15d110022a59"
Nov 02 15:14:49.077694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.077676    5041 scope.go:122] "RemoveContainer" containerID="1849ac7961a4bf0aaad9b575c04c5e3ca7726e7fafd2023bafe2118a685d1108"
Nov 02 15:14:49.087964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.087940    5041 scope.go:122] "RemoveContainer" containerID="5101c868f670a042a09e4dbe581ada8861ea1b930ad7dd91dac94a1c4ab647f5"
Nov 02 15:14:49.098956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.098933    5041 scope.go:122] "RemoveContainer" containerID="8a4fe7f09424dc5e03927f38e5678c9bdf57fb2352299c9869152813ff642d87"
Nov 02 15:14:49.110302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.110282    5041 scope.go:122] "RemoveContainer" containerID="98d3ebfe088af0d1e0e1cb9a907cb4568b127ee07aeadd9ef6483703b72bf50b"
Nov 02 15:14:49.148340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.148309    5041 scope.go:122] "RemoveContainer" containerID="57caa320cff8c80596a56dc2b24c9581b4c2d42057f8178843521cf089ab5fe9"
Nov 02 15:14:49.159355 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.159336    5041 scope.go:122] "RemoveContainer" containerID="2cddd13831f3395c35692a1c6af64ae650bf3ec8812488708cc53678611d5af7"
Nov 02 15:14:49.169865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.169843    5041 scope.go:122] "RemoveContainer" containerID="92774dce9bf0c807faa14e9f579eba3e8e9ef489587ad8d182d6b4d1c1147ceb"
Nov 02 15:14:49.179993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.179971    5041 scope.go:122] "RemoveContainer" containerID="0788a001b50585f91bd8e84401979ee1fe0d724adea3cde4abca9fb3032deaeb"
Nov 02 15:14:49.190011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.189991    5041 scope.go:122] "RemoveContainer" containerID="1385d820c5619e5611371475cfb7f6879119c5e5adc04e741a6d3a3dc5d1f85d"
Nov 02 15:14:49.202794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.202771    5041 scope.go:122] "RemoveContainer" containerID="c6958f09e551b3edd1995fec0c81aa14b18098af75deb455da5f6fbe55e4f7f6"
Nov 02 15:14:49.238756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.238591    5041 scope.go:122] "RemoveContainer" containerID="1d2e11e7da35a22523ac0ebc21b8bafa0d253f1c0691fd5cc4b9a958d4cdea7f"
Nov 02 15:14:49.538810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.538777    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 15:14:49.538990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.538830    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 15:14:49.891555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.891520    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4087f33d-f5e2-4150-ae4b-3ca5252d3162" path="/var/lib/kubelet/pods/4087f33d-f5e2-4150-ae4b-3ca5252d3162/volumes"
Nov 02 15:14:49.892153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:49.892132    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888646d2-32a2-4eb5-8ed1-63112f041266" path="/var/lib/kubelet/pods/888646d2-32a2-4eb5-8ed1-63112f041266/volumes"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.236837    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"]
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238036    5041 cpu_manager.go:414] "containerMap: removing container" podUID="888646d2-32a2-4eb5-8ed1-63112f041266" containerName="payload-pre-processing"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238059    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888646d2-32a2-4eb5-8ed1-63112f041266" containerName="payload-pre-processing"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238097    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e903add9-16cc-46f1-af47-f2f7088bddaa" containerName="revoke-keys"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238104    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e903add9-16cc-46f1-af47-f2f7088bddaa" containerName="revoke-keys"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238119    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4087f33d-f5e2-4150-ae4b-3ca5252d3162" containerName="payload-processing"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238126    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4087f33d-f5e2-4150-ae4b-3ca5252d3162" containerName="payload-processing"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238344    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e903add9-16cc-46f1-af47-f2f7088bddaa" containerName="revoke-keys"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238378    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4087f33d-f5e2-4150-ae4b-3ca5252d3162" containerName="payload-processing"
Nov 02 15:14:50.253161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.238402    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="888646d2-32a2-4eb5-8ed1-63112f041266" containerName="payload-pre-processing"
Nov 02 15:14:50.542101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.541293    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 15:14:50.542101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.541378    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 15:14:50.649385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.646881    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"]
Nov 02 15:14:50.649385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.646936    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"]
Nov 02 15:14:50.679387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.672648    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:50.717006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.716250    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-453864\""
Nov 02 15:14:50.760402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.757835    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:50.760402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.757923    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4pd\" (UniqueName: \"kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd\") pod \"payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:50.842221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.765718    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"]
Nov 02 15:14:50.842221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.765848    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:50.842221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.804622    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-453864-dockercfg-qwks7\""
Nov 02 15:14:50.884466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.862757    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77lh\" (UniqueName: \"kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh\") pod \"payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:50.884466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.862832    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:50.884466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.863139    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:50.884466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.863198    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4pd\" (UniqueName: \"kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd\") pod \"payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:50.884466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:50.864318    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:14:51.009301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.009195    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:51.082394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.032417    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4pd\" (UniqueName: \"kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd\") pod \"payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:51.082394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.055798    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:51.158223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.158185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77lh\" (UniqueName: \"kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh\") pod \"payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:51.172734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.172703    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:51.197316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.197267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:51.253413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.253345    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77lh\" (UniqueName: \"kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh\") pod \"payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:51.497238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.497205    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:51.544823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.544782    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 15:14:51.544923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.544844    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 15:14:51.549450 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:51.549421    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode617804b_9177_4740_b61e_9ae08e8f45a0.slice/crio-dda3d0262fc8a81926ffd9aa4bbcae774ac5a9a5eab2714301d1b8f44a4a7f52 WatchSource:0}: Error finding container dda3d0262fc8a81926ffd9aa4bbcae774ac5a9a5eab2714301d1b8f44a4a7f52: Status 404 returned error can't find the container with id dda3d0262fc8a81926ffd9aa4bbcae774ac5a9a5eab2714301d1b8f44a4a7f52
Nov 02 15:14:51.551467 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.551440    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"]
Nov 02 15:14:51.845628 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:51.845598    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66419e73_7d9e_45f8_bf42_d38f673f4d7b.slice/crio-8a33a269daaf9c25ce8379def879afb69e279f0c502104b8a6fdce8b557df586 WatchSource:0}: Error finding container 8a33a269daaf9c25ce8379def879afb69e279f0c502104b8a6fdce8b557df586: Status 404 returned error can't find the container with id 8a33a269daaf9c25ce8379def879afb69e279f0c502104b8a6fdce8b557df586
Nov 02 15:14:51.846884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:51.846860    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"]
Nov 02 15:14:52.033443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.033406    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" event={"ID":"66419e73-7d9e-45f8-bf42-d38f673f4d7b","Type":"ContainerStarted","Data":"679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009"}
Nov 02 15:14:52.033854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.033460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" event={"ID":"66419e73-7d9e-45f8-bf42-d38f673f4d7b","Type":"ContainerStarted","Data":"8a33a269daaf9c25ce8379def879afb69e279f0c502104b8a6fdce8b557df586"}
Nov 02 15:14:52.033854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.033492    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:14:52.035548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.035517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" event={"ID":"e617804b-9177-4740-b61e-9ae08e8f45a0","Type":"ContainerStarted","Data":"9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba"}
Nov 02 15:14:52.035672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.035576    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:14:52.035672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.035606    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" event={"ID":"e617804b-9177-4740-b61e-9ae08e8f45a0","Type":"ContainerStarted","Data":"dda3d0262fc8a81926ffd9aa4bbcae774ac5a9a5eab2714301d1b8f44a4a7f52"}
Nov 02 15:14:52.071534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.071467    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" podStartSLOduration=2.071448163 podStartE2EDuration="2.071448163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:52.069235579 +0000 UTC m=+1144.327305029" watchObservedRunningTime="2026-11-02 15:14:52.071448163 +0000 UTC m=+1144.329517614"
Nov 02 15:14:52.106544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.106449    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" podStartSLOduration=2.106435412 podStartE2EDuration="2.106435412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:52.10495879 +0000 UTC m=+1144.363028240" watchObservedRunningTime="2026-11-02 15:14:52.106435412 +0000 UTC m=+1144.364504861"
Nov 02 15:14:52.322412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.319569    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"]
Nov 02 15:14:52.322412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.319862    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl" podUID="bfef5470-e65d-4bc7-8332-38c069a37c1c" containerName="maas-api" containerID="cri-o://9174af792951a1e5a2fbcd34f5fb5fb894a6b89151733682a15966a17889c470" gracePeriod=30
Nov 02 15:14:52.651440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.637481    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:14:52.651440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:52.640943    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:14:53.180007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.179717    5041 generic.go:356] "Generic (PLEG): container finished" podID="bfef5470-e65d-4bc7-8332-38c069a37c1c" containerID="9174af792951a1e5a2fbcd34f5fb5fb894a6b89151733682a15966a17889c470" exitCode=0
Nov 02 15:14:53.180007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.179781    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl" event={"ID":"bfef5470-e65d-4bc7-8332-38c069a37c1c","Type":"ContainerDied","Data":"9174af792951a1e5a2fbcd34f5fb5fb894a6b89151733682a15966a17889c470"}
Nov 02 15:14:53.290828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.290796    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:14:53.305585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.305555    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs\") pod \"bfef5470-e65d-4bc7-8332-38c069a37c1c\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") "
Nov 02 15:14:53.305736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.305717    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls\") pod \"bfef5470-e65d-4bc7-8332-38c069a37c1c\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") "
Nov 02 15:14:53.305877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.305863    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca\") pod \"bfef5470-e65d-4bc7-8332-38c069a37c1c\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") "
Nov 02 15:14:53.305948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.305932    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk\") pod \"bfef5470-e65d-4bc7-8332-38c069a37c1c\" (UID: \"bfef5470-e65d-4bc7-8332-38c069a37c1c\") "
Nov 02 15:14:53.306402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.306357    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca" pod "bfef5470-e65d-4bc7-8332-38c069a37c1c" (UID: "bfef5470-e65d-4bc7-8332-38c069a37c1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:14:53.306663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.306644    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef5470-e65d-4bc7-8332-38c069a37c1c-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:53.307790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.307764    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls" pod "bfef5470-e65d-4bc7-8332-38c069a37c1c" (UID: "bfef5470-e65d-4bc7-8332-38c069a37c1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:14:53.307790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.307780    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs" pod "bfef5470-e65d-4bc7-8332-38c069a37c1c" (UID: "bfef5470-e65d-4bc7-8332-38c069a37c1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:14:53.308048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.308027    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk" pod "bfef5470-e65d-4bc7-8332-38c069a37c1c" (UID: "bfef5470-e65d-4bc7-8332-38c069a37c1c"). InnerVolumeSpecName "kube-api-access-f4ckk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:14:53.319769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.319751    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"]
Nov 02 15:14:53.325726 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.325675    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-991657be-4ff4p"]
Nov 02 15:14:53.454463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.410794    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:53.454627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.454492    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ckk\" (UniqueName: \"kubernetes.io/projected/bfef5470-e65d-4bc7-8332-38c069a37c1c-kube-api-access-f4ckk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:53.454627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.454521    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfef5470-e65d-4bc7-8332-38c069a37c1c-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:14:53.890374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:53.890334    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e903add9-16cc-46f1-af47-f2f7088bddaa" path="/var/lib/kubelet/pods/e903add9-16cc-46f1-af47-f2f7088bddaa/volumes"
Nov 02 15:14:54.291860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:54.291783    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl" event={"ID":"bfef5470-e65d-4bc7-8332-38c069a37c1c","Type":"ContainerDied","Data":"dc18ded47cc44a57e163a43a713f7fc7adac2d5c415ea09b111dc6d35d604f8e"}
Nov 02 15:14:54.292583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:54.292453    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"
Nov 02 15:14:54.297463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:54.297412    5041 scope.go:122] "RemoveContainer" containerID="9174af792951a1e5a2fbcd34f5fb5fb894a6b89151733682a15966a17889c470"
Nov 02 15:14:54.367817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:54.367745    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"]
Nov 02 15:14:54.394978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:54.394920    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-991657be-5667d6d7f7-jblvl"]
Nov 02 15:14:55.387974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.387910    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"]
Nov 02 15:14:55.389785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.389755    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bfef5470-e65d-4bc7-8332-38c069a37c1c" containerName="maas-api"
Nov 02 15:14:55.389946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.389931    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfef5470-e65d-4bc7-8332-38c069a37c1c" containerName="maas-api"
Nov 02 15:14:55.390235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.390220    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfef5470-e65d-4bc7-8332-38c069a37c1c" containerName="maas-api"
Nov 02 15:14:55.437388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.428423    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"]
Nov 02 15:14:55.437782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.437744    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.463292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.463253    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fda646-openshift-default-dockercfg-kdct4\""
Nov 02 15:14:55.600766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600382    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.600766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600498    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.600766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600525    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.600766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600697    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.601112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600776    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.601112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600805    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.601112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600853    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d7c\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.601112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600960    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.601112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.600992    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702404    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702447    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702543    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702615    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702634    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702749    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702821    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702893    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d7c\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.702990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.702932    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.703251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.703225    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.703393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.703261    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.703393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.703285    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.703504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.703460    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.705113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.705094    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.705357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.705340    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.723747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.723712    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d7c\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.724724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.724409    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.948630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.948598    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:55.992271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:55.990579    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfef5470-e65d-4bc7-8332-38c069a37c1c" path="/var/lib/kubelet/pods/bfef5470-e65d-4bc7-8332-38c069a37c1c/volumes"
Nov 02 15:14:56.326058 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:14:56.326023    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b2c4b5_e9c2_4e7d_95f3_a2240f54eeb0.slice/crio-ef46db6ba112d2a7c478857d13fa513f6114e5cd3100b9732a914859aeb2266c WatchSource:0}: Error finding container ef46db6ba112d2a7c478857d13fa513f6114e5cd3100b9732a914859aeb2266c: Status 404 returned error can't find the container with id ef46db6ba112d2a7c478857d13fa513f6114e5cd3100b9732a914859aeb2266c
Nov 02 15:14:56.328602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.328577    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"]
Nov 02 15:14:56.328959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.328923    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:56.329044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.328994    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:56.329044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.329033    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:56.641191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.641105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" event={"ID":"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0","Type":"ContainerStarted","Data":"dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"}
Nov 02 15:14:56.641191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.641165    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" event={"ID":"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0","Type":"ContainerStarted","Data":"ef46db6ba112d2a7c478857d13fa513f6114e5cd3100b9732a914859aeb2266c"}
Nov 02 15:14:56.682538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.682481    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podStartSLOduration=1.68246344 podStartE2EDuration="1.68246344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:56.681316927 +0000 UTC m=+1148.939386377" watchObservedRunningTime="2026-11-02 15:14:56.68246344 +0000 UTC m=+1148.940532892"
Nov 02 15:14:56.959861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:56.959821    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:58.046423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:58.045351    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:58.846005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:58.844853    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:14:58.847873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:14:58.847555    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:15:00.183386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.175323    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"]
Nov 02 15:15:00.183386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.181661    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.211454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.211429    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:15:00.220837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.220724    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"]
Nov 02 15:15:00.230253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.230230    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.302788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.269450    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"]
Nov 02 15:15:00.340494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.340465    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.369656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.369628    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjff\" (UniqueName: \"kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.369925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.369901    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.382615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.382580    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"]
Nov 02 15:15:00.397313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.397286    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"]
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.475134    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjff\" (UniqueName: \"kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.475225    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.475469    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvcg\" (UniqueName: \"kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.475528    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.475593    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.475709    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnbd\" (UniqueName: \"kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.488194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.476986    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.580501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.577942    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvcg\" (UniqueName: \"kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.580501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.578028    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.580501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.578086    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.580501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.578198    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnbd\" (UniqueName: \"kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.580501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.579671    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.580501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.580453    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.589822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.589521    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"]
Nov 02 15:15:00.589822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.589570    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"]
Nov 02 15:15:00.590043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.590016    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjff\" (UniqueName: \"kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.594140 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.594110    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.596935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.596845    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"]
Nov 02 15:15:00.597060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.596943    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.637423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.625493    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnbd\" (UniqueName: \"kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.637423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.627532    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"]
Nov 02 15:15:00.647492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.647414    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"]
Nov 02 15:15:00.665390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.662645    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:00.685395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.682171    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpljs\" (UniqueName: \"kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.685395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.682305    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.685395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.682441    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.685395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.682618    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4sk\" (UniqueName: \"kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.693384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.690023    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvcg\" (UniqueName: \"kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:00.796384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.794181    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4sk\" (UniqueName: \"kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.796384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.794325    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpljs\" (UniqueName: \"kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.796384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.794457    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.796384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.794577    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.796384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.795547    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.800387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.797938    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.827472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.827446    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:00.833220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.833194    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4sk\" (UniqueName: \"kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:00.834016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.833992    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpljs\" (UniqueName: \"kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:00.890245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:00.890220    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:01.065640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.065044    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:01.072454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.069927    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:01.442917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.442884    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"]
Nov 02 15:15:01.614919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.614885    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"]
Nov 02 15:15:01.722977 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:01.722949    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1711e30_02c3_4008_8bf9_deeb2c89ccf3.slice/crio-d1799a3d232a753626e13ae12261becb1b348ecb63b14e2f7c68071fb1fadf7f WatchSource:0}: Error finding container d1799a3d232a753626e13ae12261becb1b348ecb63b14e2f7c68071fb1fadf7f: Status 404 returned error can't find the container with id d1799a3d232a753626e13ae12261becb1b348ecb63b14e2f7c68071fb1fadf7f
Nov 02 15:15:01.732198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.728612    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"]
Nov 02 15:15:01.736712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.735576    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:01.738565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.738472    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"]
Nov 02 15:15:01.747382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.747341    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"]
Nov 02 15:15:01.772475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.772421    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fda646\""
Nov 02 15:15:01.772867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.772849    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fda646-dockercfg-6xxwb\""
Nov 02 15:15:01.790087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.790042    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"]
Nov 02 15:15:01.958185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.838032    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:01.958185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.838117    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7pc\" (UniqueName: \"kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc\") pod \"payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:01.958185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.947559    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:01.958185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.947739    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7pc\" (UniqueName: \"kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc\") pod \"payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:01.958185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:01.949216    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:02.256719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.256623    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"]
Nov 02 15:15:02.266541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.266506    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerStarted","Data":"ac4ee80cf7e9043391858ce27816ceb94da2ae0cf30fda7d883723032a8b73e9"}
Nov 02 15:15:02.266772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.266729    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.268578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.266768    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerStarted","Data":"56c886153ff3db74b4d5317812ab80ecd3e22874ee8aff9a8b39d00942da5bfe"}
Nov 02 15:15:02.269204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.268717    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerStarted","Data":"35fa89bb34b21f8557548df1623108c23f426eced735d8b421940796725aa3af"}
Nov 02 15:15:02.269204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.268753    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerStarted","Data":"d1799a3d232a753626e13ae12261becb1b348ecb63b14e2f7c68071fb1fadf7f"}
Nov 02 15:15:02.275070 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.275049    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"]
Nov 02 15:15:02.282392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.282354    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"]
Nov 02 15:15:02.283072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.283048    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7pc\" (UniqueName: \"kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc\") pod \"payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:02.392206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.391587    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkp2\" (UniqueName: \"kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2\") pod \"payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.392206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.391815    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.457239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.457207    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:02.493219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.493186    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkp2\" (UniqueName: \"kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2\") pod \"payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.493397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.493375    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.494013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.493993    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.634688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.633166    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkp2\" (UniqueName: \"kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2\") pod \"payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.728611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.728580    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:02.902242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:02.902209    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"]
Nov 02 15:15:03.187740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.187423    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:15:03.188751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.188403    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:15:03.318512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.317726    5041 generic.go:356] "Generic (PLEG): container finished" podID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerID="9fcb319a822f94a5764957ef76f89af55876d3220322ef19a3ef36f93681de76" exitCode=60
Nov 02 15:15:03.318512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.317842    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerDied","Data":"9fcb319a822f94a5764957ef76f89af55876d3220322ef19a3ef36f93681de76"}
Nov 02 15:15:03.318512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.318138    5041 scope.go:122] "RemoveContainer" containerID="9fcb319a822f94a5764957ef76f89af55876d3220322ef19a3ef36f93681de76"
Nov 02 15:15:03.325816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.324962    5041 generic.go:356] "Generic (PLEG): container finished" podID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerID="e6e7912bfd0e3c4ea1d4321f0b9368196598c85a797ebd195b6205b5c9a0690d" exitCode=60
Nov 02 15:15:03.325816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.325117    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerDied","Data":"e6e7912bfd0e3c4ea1d4321f0b9368196598c85a797ebd195b6205b5c9a0690d"}
Nov 02 15:15:03.325816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.325469    5041 scope.go:122] "RemoveContainer" containerID="e6e7912bfd0e3c4ea1d4321f0b9368196598c85a797ebd195b6205b5c9a0690d"
Nov 02 15:15:03.335842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.334612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerStarted","Data":"7b6e7388a243292fef23373240fd86129055c98e1690edf318b05cce7a4d0d95"}
Nov 02 15:15:03.339029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.338510    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerStarted","Data":"06167abdeb338433a45d6a307008a3a731fa669f6795814ba7fa4c4f3c500913"}
Nov 02 15:15:03.339029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.338564    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerStarted","Data":"3d36c272262e8c40c317fbb9ce0c4b7cb19dc57b87b9725c7486de04dc3414c2"}
Nov 02 15:15:03.341444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.341342    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" event={"ID":"502b3812-a75a-46ca-b156-2189ef4c8d29","Type":"ContainerStarted","Data":"7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d"}
Nov 02 15:15:03.341444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.341408    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" event={"ID":"502b3812-a75a-46ca-b156-2189ef4c8d29","Type":"ContainerStarted","Data":"7aa0345349775e85e3226a4fdeb1dc898c576a9b27e8109cf9e3184286f03754"}
Nov 02 15:15:03.341788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.341765    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:03.343449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.343425    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerStarted","Data":"b89f34d40698e4233ef2c71fe6e67a0cac3e100dff0104c269ce660d5828647e"}
Nov 02 15:15:03.359126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.359094    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"]
Nov 02 15:15:03.359699 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:03.359671    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61a32ad_3972_472b_807a_a9ec6a6bcd6b.slice/crio-2e04e09312c62b8b014721874ba9595c9a1ce79423764bbaa9c1fbe1286280c6 WatchSource:0}: Error finding container 2e04e09312c62b8b014721874ba9595c9a1ce79423764bbaa9c1fbe1286280c6: Status 404 returned error can't find the container with id 2e04e09312c62b8b014721874ba9595c9a1ce79423764bbaa9c1fbe1286280c6
Nov 02 15:15:03.471466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.470503    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" podStartSLOduration=3.4704842879999998 podStartE2EDuration="3.470484288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.444774666 +0000 UTC m=+1155.702844323" watchObservedRunningTime="2026-11-02 15:15:03.470484288 +0000 UTC m=+1155.728553738"
Nov 02 15:15:03.499216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.499166    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" podStartSLOduration=3.4991532850000002 podStartE2EDuration="3.499153285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.488019624 +0000 UTC m=+1155.746089075" watchObservedRunningTime="2026-11-02 15:15:03.499153285 +0000 UTC m=+1155.757222755"
Nov 02 15:15:03.532614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.532562    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" podStartSLOduration=2.532548751 podStartE2EDuration="2.532548751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.526970466 +0000 UTC m=+1155.785039918" watchObservedRunningTime="2026-11-02 15:15:03.532548751 +0000 UTC m=+1155.790618201"
Nov 02 15:15:03.565935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:03.565879    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" podStartSLOduration=3.565865923 podStartE2EDuration="3.565865923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.562178346 +0000 UTC m=+1155.820247823" watchObservedRunningTime="2026-11-02 15:15:03.565865923 +0000 UTC m=+1155.823935372"
Nov 02 15:15:04.358429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.358294    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" event={"ID":"a61a32ad-3972-472b-807a-a9ec6a6bcd6b","Type":"ContainerStarted","Data":"1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321"}
Nov 02 15:15:04.358429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.358376    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" event={"ID":"a61a32ad-3972-472b-807a-a9ec6a6bcd6b","Type":"ContainerStarted","Data":"2e04e09312c62b8b014721874ba9595c9a1ce79423764bbaa9c1fbe1286280c6"}
Nov 02 15:15:04.358641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.358439    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:04.360475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.360448    5041 generic.go:356] "Generic (PLEG): container finished" podID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerID="7b6e7388a243292fef23373240fd86129055c98e1690edf318b05cce7a4d0d95" exitCode=60
Nov 02 15:15:04.360591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.360529    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerDied","Data":"7b6e7388a243292fef23373240fd86129055c98e1690edf318b05cce7a4d0d95"}
Nov 02 15:15:04.360868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.360840    5041 scope.go:122] "RemoveContainer" containerID="7b6e7388a243292fef23373240fd86129055c98e1690edf318b05cce7a4d0d95"
Nov 02 15:15:04.362484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.362462    5041 generic.go:356] "Generic (PLEG): container finished" podID="b5075de1-a46e-4c77-9098-58f98918d782" containerID="06167abdeb338433a45d6a307008a3a731fa669f6795814ba7fa4c4f3c500913" exitCode=60
Nov 02 15:15:04.362595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.362577    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerDied","Data":"06167abdeb338433a45d6a307008a3a731fa669f6795814ba7fa4c4f3c500913"}
Nov 02 15:15:04.362885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.362869    5041 scope.go:122] "RemoveContainer" containerID="06167abdeb338433a45d6a307008a3a731fa669f6795814ba7fa4c4f3c500913"
Nov 02 15:15:04.366317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.366177    5041 generic.go:356] "Generic (PLEG): container finished" podID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerID="b89f34d40698e4233ef2c71fe6e67a0cac3e100dff0104c269ce660d5828647e" exitCode=60
Nov 02 15:15:04.366317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.366184    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerDied","Data":"b89f34d40698e4233ef2c71fe6e67a0cac3e100dff0104c269ce660d5828647e"}
Nov 02 15:15:04.366521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.366385    5041 scope.go:122] "RemoveContainer" containerID="b89f34d40698e4233ef2c71fe6e67a0cac3e100dff0104c269ce660d5828647e"
Nov 02 15:15:04.368751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.368730    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerStarted","Data":"60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390"}
Nov 02 15:15:04.371599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.371574    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerStarted","Data":"d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f"}
Nov 02 15:15:04.398609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.398557    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" podStartSLOduration=3.3985401250000002 podStartE2EDuration="3.398540125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:04.394649745 +0000 UTC m=+1156.652719199" watchObservedRunningTime="2026-11-02 15:15:04.398540125 +0000 UTC m=+1156.656609576"
Nov 02 15:15:04.587392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.578058    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" podStartSLOduration=4.578001073 podStartE2EDuration="4.578001073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:04.549904731 +0000 UTC m=+1156.807974182" watchObservedRunningTime="2026-11-02 15:15:04.578001073 +0000 UTC m=+1156.836070524"
Nov 02 15:15:04.676400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:04.649495    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" podStartSLOduration=4.649470061 podStartE2EDuration="4.649470061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:04.640643046 +0000 UTC m=+1156.898712496" watchObservedRunningTime="2026-11-02 15:15:04.649470061 +0000 UTC m=+1156.907539535"
Nov 02 15:15:05.387173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.387082    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerStarted","Data":"f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6"}
Nov 02 15:15:05.389907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.389875    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerStarted","Data":"7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038"}
Nov 02 15:15:05.392090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.392059    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerStarted","Data":"1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed"}
Nov 02 15:15:05.394374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.394331    5041 generic.go:356] "Generic (PLEG): container finished" podID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerID="60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390" exitCode=60
Nov 02 15:15:05.394464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.394398    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerDied","Data":"60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390"}
Nov 02 15:15:05.394539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.394476    5041 scope.go:122] "RemoveContainer" containerID="9fcb319a822f94a5764957ef76f89af55876d3220322ef19a3ef36f93681de76"
Nov 02 15:15:05.394744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.394719    5041 scope.go:122] "RemoveContainer" containerID="60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390"
Nov 02 15:15:05.395020 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:05.394994    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj_odh-ai-gateway-infra(e6d847d0-1640-433a-8b17-71ab99a033a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3"
Nov 02 15:15:05.397770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.397744    5041 generic.go:356] "Generic (PLEG): container finished" podID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerID="d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f" exitCode=60
Nov 02 15:15:05.397924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.397897    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerDied","Data":"d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f"}
Nov 02 15:15:05.398041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.398030    5041 scope.go:122] "RemoveContainer" containerID="d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f"
Nov 02 15:15:05.398349 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:05.398317    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4_odh-ai-gateway-infra(90527bb2-3077-49bb-be87-9bffcd0c5caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa"
Nov 02 15:15:05.409923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:05.409898    5041 scope.go:122] "RemoveContainer" containerID="e6e7912bfd0e3c4ea1d4321f0b9368196598c85a797ebd195b6205b5c9a0690d"
Nov 02 15:15:06.440303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.440270    5041 generic.go:356] "Generic (PLEG): container finished" podID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerID="1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed" exitCode=60
Nov 02 15:15:06.440790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.440355    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerDied","Data":"1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed"}
Nov 02 15:15:06.440790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.440441    5041 scope.go:122] "RemoveContainer" containerID="b89f34d40698e4233ef2c71fe6e67a0cac3e100dff0104c269ce660d5828647e"
Nov 02 15:15:06.440925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.440816    5041 scope.go:122] "RemoveContainer" containerID="1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed"
Nov 02 15:15:06.441240 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:06.441215    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz_odh-ai-gateway-infra(d8a6b98a-106a-4aeb-a72b-b410a86e9e71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71"
Nov 02 15:15:06.443036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.443010    5041 scope.go:122] "RemoveContainer" containerID="60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390"
Nov 02 15:15:06.443488 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:06.443460    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj_odh-ai-gateway-infra(e6d847d0-1640-433a-8b17-71ab99a033a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3"
Nov 02 15:15:06.444927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.444907    5041 scope.go:122] "RemoveContainer" containerID="d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f"
Nov 02 15:15:06.445202 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:06.445177    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4_odh-ai-gateway-infra(90527bb2-3077-49bb-be87-9bffcd0c5caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa"
Nov 02 15:15:06.447175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.447148    5041 generic.go:356] "Generic (PLEG): container finished" podID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerID="f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6" exitCode=60
Nov 02 15:15:06.447251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.447229    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerDied","Data":"f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6"}
Nov 02 15:15:06.447453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.447433    5041 scope.go:122] "RemoveContainer" containerID="f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6"
Nov 02 15:15:06.447781 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:06.447761    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587_odh-ai-gateway-infra(a1711e30-02c3-4008-8bf9-deeb2c89ccf3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3"
Nov 02 15:15:06.449484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.449461    5041 generic.go:356] "Generic (PLEG): container finished" podID="b5075de1-a46e-4c77-9098-58f98918d782" containerID="7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038" exitCode=60
Nov 02 15:15:06.449628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.449607    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerDied","Data":"7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038"}
Nov 02 15:15:06.449773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.449758    5041 scope.go:122] "RemoveContainer" containerID="7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038"
Nov 02 15:15:06.450028 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:06.450010    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw_odh-ai-gateway-infra(b5075de1-a46e-4c77-9098-58f98918d782)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" podUID="b5075de1-a46e-4c77-9098-58f98918d782"
Nov 02 15:15:06.458785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.458761    5041 scope.go:122] "RemoveContainer" containerID="7b6e7388a243292fef23373240fd86129055c98e1690edf318b05cce7a4d0d95"
Nov 02 15:15:06.472704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:06.472680    5041 scope.go:122] "RemoveContainer" containerID="06167abdeb338433a45d6a307008a3a731fa669f6795814ba7fa4c4f3c500913"
Nov 02 15:15:06.955927 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:06.955881    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:15:07.461794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:07.461767    5041 scope.go:122] "RemoveContainer" containerID="1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed"
Nov 02 15:15:07.462254 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:07.462225    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz_odh-ai-gateway-infra(d8a6b98a-106a-4aeb-a72b-b410a86e9e71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71"
Nov 02 15:15:07.464730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:07.464665    5041 scope.go:122] "RemoveContainer" containerID="f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6"
Nov 02 15:15:07.465095 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:07.465068    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587_odh-ai-gateway-infra(a1711e30-02c3-4008-8bf9-deeb2c89ccf3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3"
Nov 02 15:15:07.466706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:07.466685    5041 scope.go:122] "RemoveContainer" containerID="7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038"
Nov 02 15:15:07.466899 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:07.466880    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw_odh-ai-gateway-infra(b5075de1-a46e-4c77-9098-58f98918d782)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" podUID="b5075de1-a46e-4c77-9098-58f98918d782"
Nov 02 15:15:08.342953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.342918    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"]
Nov 02 15:15:08.348475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.348448    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.387884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.377733    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-453864-serving-cert\""
Nov 02 15:15:08.387884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.387511    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"]
Nov 02 15:15:08.519958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.519923    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.520307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.519973    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.520307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.520170    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhn4\" (UniqueName: \"kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.520307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.520208    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.627048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.626944    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhn4\" (UniqueName: \"kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.636132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.636078    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.636990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.636955    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.637211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.637191    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.650856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.649875    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.697450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.689026    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.697450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.690210    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.697450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.695460    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhn4\" (UniqueName: \"kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4\") pod \"maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:08.966751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:08.966717    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:09.300912 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:09.300872    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0714ad_acf9_4929_a875_81c3b757d8b4.slice/crio-adaa8db17ae1361a5df654e4b8e7dbfa5b37637bdd54d6c8f44489ac52be99c4 WatchSource:0}: Error finding container adaa8db17ae1361a5df654e4b8e7dbfa5b37637bdd54d6c8f44489ac52be99c4: Status 404 returned error can't find the container with id adaa8db17ae1361a5df654e4b8e7dbfa5b37637bdd54d6c8f44489ac52be99c4
Nov 02 15:15:09.304636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:09.304610    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"]
Nov 02 15:15:09.644239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:09.644116    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc" event={"ID":"6e0714ad-acf9-4929-a875-81c3b757d8b4","Type":"ContainerStarted","Data":"adaa8db17ae1361a5df654e4b8e7dbfa5b37637bdd54d6c8f44489ac52be99c4"}
Nov 02 15:15:10.661121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:10.661081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc" event={"ID":"6e0714ad-acf9-4929-a875-81c3b757d8b4","Type":"ContainerStarted","Data":"4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88"}
Nov 02 15:15:10.661544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:10.661171    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:10.703229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:10.703183    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc" podStartSLOduration=2.262868744 podStartE2EDuration="2.703169928s" totalImagesPullingTime="440.301184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:10.700155803 +0000 UTC m=+1162.958225252" watchObservedRunningTime="2026-11-02 15:15:10.703169928 +0000 UTC m=+1162.961239378"
Nov 02 15:15:14.375284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:14.375254    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:15:15.433384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:15.428447    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:15:16.676491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:16.676459    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:15:16.880340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:16.880312    5041 scope.go:122] "RemoveContainer" containerID="60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390"
Nov 02 15:15:17.798091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:17.798055    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerStarted","Data":"74a375c839a0a85d760d4a1be2b1c6eaf71a582b05399aa95fe55fd83e60775b"}
Nov 02 15:15:17.889506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:17.889476    5041 scope.go:122] "RemoveContainer" containerID="7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038"
Nov 02 15:15:18.416814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.416731    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"]
Nov 02 15:15:18.421236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.421211    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.437999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.437973    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-fda646-serving-cert\""
Nov 02 15:15:18.454241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.454212    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"]
Nov 02 15:15:18.495741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.495701    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.495898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.495835    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.496056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.496038    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wgt\" (UniqueName: \"kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.496257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.496235    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.598266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.598235    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wgt\" (UniqueName: \"kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.598449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.598351    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.598543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.598446    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.598543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.598513    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.599094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.599067    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.600908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.600883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.601133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.600983    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.619229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.619207    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wgt\" (UniqueName: \"kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt\") pod \"maas-api-e2e-shared-a-w1-fda646-864497c546-296k5\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.753762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.749998    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:18.884095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.884058    5041 scope.go:122] "RemoveContainer" containerID="f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6"
Nov 02 15:15:18.886676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.886654    5041 scope.go:122] "RemoveContainer" containerID="d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f"
Nov 02 15:15:18.898477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.898455    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"]
Nov 02 15:15:18.914057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.914022    5041 generic.go:356] "Generic (PLEG): container finished" podID="b5075de1-a46e-4c77-9098-58f98918d782" containerID="1e786a60d920fe3159e0798713c3702ab811c143c0b9042381cd2b8fbf93077d" exitCode=60
Nov 02 15:15:18.914423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.914303    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerDied","Data":"1e786a60d920fe3159e0798713c3702ab811c143c0b9042381cd2b8fbf93077d"}
Nov 02 15:15:18.914423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.914416    5041 scope.go:122] "RemoveContainer" containerID="7011dc4731ec70d614341a100692b4830616dd16cdd396cbfe0cff8ad7335038"
Nov 02 15:15:18.914906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.914886    5041 scope.go:122] "RemoveContainer" containerID="1e786a60d920fe3159e0798713c3702ab811c143c0b9042381cd2b8fbf93077d"
Nov 02 15:15:18.915317 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:18.915289    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw_odh-ai-gateway-infra(b5075de1-a46e-4c77-9098-58f98918d782)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" podUID="b5075de1-a46e-4c77-9098-58f98918d782"
Nov 02 15:15:18.918639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.918618    5041 generic.go:356] "Generic (PLEG): container finished" podID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerID="74a375c839a0a85d760d4a1be2b1c6eaf71a582b05399aa95fe55fd83e60775b" exitCode=60
Nov 02 15:15:18.918777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.918752    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerDied","Data":"74a375c839a0a85d760d4a1be2b1c6eaf71a582b05399aa95fe55fd83e60775b"}
Nov 02 15:15:18.999680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:18.999582    5041 scope.go:122] "RemoveContainer" containerID="60a9bb8a81c4770622e55c82f8ca2ee98f6b9f27c3d07a42f7e132001d1b4390"
Nov 02 15:15:19.178973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.178948    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:19.210805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.210775    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"]
Nov 02 15:15:19.210914 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:19.210831    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd10f188_3b57_4220_a11d_33972f7b069f.slice/crio-9e1e925b01dedc153b8a5fab40873b84aae6a69a4bd7d205ba73883c3af296a0 WatchSource:0}: Error finding container 9e1e925b01dedc153b8a5fab40873b84aae6a69a4bd7d205ba73883c3af296a0: Status 404 returned error can't find the container with id 9e1e925b01dedc153b8a5fab40873b84aae6a69a4bd7d205ba73883c3af296a0
Nov 02 15:15:19.247424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.247393    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca\") pod \"e6d847d0-1640-433a-8b17-71ab99a033a3\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") "
Nov 02 15:15:19.247609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.247591    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff\" (UniqueName: \"kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff\") pod \"e6d847d0-1640-433a-8b17-71ab99a033a3\" (UID: \"e6d847d0-1640-433a-8b17-71ab99a033a3\") "
Nov 02 15:15:19.249135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.249090    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca" pod "e6d847d0-1640-433a-8b17-71ab99a033a3" (UID: "e6d847d0-1640-433a-8b17-71ab99a033a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:19.251393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.251349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff" pod "e6d847d0-1640-433a-8b17-71ab99a033a3" (UID: "e6d847d0-1640-433a-8b17-71ab99a033a3"). InnerVolumeSpecName "kube-api-access-wjjff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:19.350104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.350069    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d847d0-1640-433a-8b17-71ab99a033a3-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:19.350104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:19.350097    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjjff\" (UniqueName: \"kubernetes.io/projected/e6d847d0-1640-433a-8b17-71ab99a033a3-kube-api-access-wjjff\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:20.007784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.007191    5041 scope.go:122] "RemoveContainer" containerID="1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed"
Nov 02 15:15:20.014066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.014028    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"]
Nov 02 15:15:20.014204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.014111    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5" event={"ID":"bd10f188-3b57-4220-a11d-33972f7b069f","Type":"ContainerStarted","Data":"cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e"}
Nov 02 15:15:20.014204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.014145    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5" event={"ID":"bd10f188-3b57-4220-a11d-33972f7b069f","Type":"ContainerStarted","Data":"9e1e925b01dedc153b8a5fab40873b84aae6a69a4bd7d205ba73883c3af296a0"}
Nov 02 15:15:20.015544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.015513    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj" event={"ID":"e6d847d0-1640-433a-8b17-71ab99a033a3","Type":"ContainerDied","Data":"56c886153ff3db74b4d5317812ab80ecd3e22874ee8aff9a8b39d00942da5bfe"}
Nov 02 15:15:20.015659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.015572    5041 scope.go:122] "RemoveContainer" containerID="74a375c839a0a85d760d4a1be2b1c6eaf71a582b05399aa95fe55fd83e60775b"
Nov 02 15:15:20.016232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.016158    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"
Nov 02 15:15:20.016435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.016414    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:20.032298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.032262    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerStarted","Data":"ea67e6fcf07f1febf4930a3137b23c7067abbddadc4de7d4746548d617a71f8b"}
Nov 02 15:15:20.037226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.037201    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerStarted","Data":"fc1faea123ed12e627efa302b40b5dd2878f8c7b2bab46bdfc7b12a4f4814e90"}
Nov 02 15:15:20.075143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.075079    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5" podStartSLOduration=1.725228065 podStartE2EDuration="2.075060179s" totalImagesPullingTime="349.832114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:20.062538184 +0000 UTC m=+1172.320607634" watchObservedRunningTime="2026-11-02 15:15:20.075060179 +0000 UTC m=+1172.333129623"
Nov 02 15:15:20.368628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.368594    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"]
Nov 02 15:15:20.377101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.377072    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893875-n5zcj"]
Nov 02 15:15:20.386334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.386307    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:20.428647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.428621    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca\") pod \"b5075de1-a46e-4c77-9098-58f98918d782\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") "
Nov 02 15:15:20.428835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.428817    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk\" (UniqueName: \"kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk\") pod \"b5075de1-a46e-4c77-9098-58f98918d782\" (UID: \"b5075de1-a46e-4c77-9098-58f98918d782\") "
Nov 02 15:15:20.429018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.428990    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca" pod "b5075de1-a46e-4c77-9098-58f98918d782" (UID: "b5075de1-a46e-4c77-9098-58f98918d782"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:20.429532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.429513    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5075de1-a46e-4c77-9098-58f98918d782-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:20.431190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.431165    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk" pod "b5075de1-a46e-4c77-9098-58f98918d782" (UID: "b5075de1-a46e-4c77-9098-58f98918d782"). InnerVolumeSpecName "kube-api-access-qz4sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:20.530777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:20.530750    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz4sk\" (UniqueName: \"kubernetes.io/projected/b5075de1-a46e-4c77-9098-58f98918d782-kube-api-access-qz4sk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:21.052609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.052579    5041 generic.go:356] "Generic (PLEG): container finished" podID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerID="ea67e6fcf07f1febf4930a3137b23c7067abbddadc4de7d4746548d617a71f8b" exitCode=60
Nov 02 15:15:21.053088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.052647    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerDied","Data":"ea67e6fcf07f1febf4930a3137b23c7067abbddadc4de7d4746548d617a71f8b"}
Nov 02 15:15:21.053088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.052703    5041 scope.go:122] "RemoveContainer" containerID="d569e054672d3a16bb89053885f378b1ec59d2fd446a06a8b500ee4bef59098f"
Nov 02 15:15:21.053088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.053024    5041 scope.go:122] "RemoveContainer" containerID="ea67e6fcf07f1febf4930a3137b23c7067abbddadc4de7d4746548d617a71f8b"
Nov 02 15:15:21.053342 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:21.053323    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4_odh-ai-gateway-infra(90527bb2-3077-49bb-be87-9bffcd0c5caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa"
Nov 02 15:15:21.054869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.054847    5041 generic.go:356] "Generic (PLEG): container finished" podID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerID="fc1faea123ed12e627efa302b40b5dd2878f8c7b2bab46bdfc7b12a4f4814e90" exitCode=60
Nov 02 15:15:21.054962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.054917    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerDied","Data":"fc1faea123ed12e627efa302b40b5dd2878f8c7b2bab46bdfc7b12a4f4814e90"}
Nov 02 15:15:21.055219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.055200    5041 scope.go:122] "RemoveContainer" containerID="fc1faea123ed12e627efa302b40b5dd2878f8c7b2bab46bdfc7b12a4f4814e90"
Nov 02 15:15:21.055484 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:21.055461    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587_odh-ai-gateway-infra(a1711e30-02c3-4008-8bf9-deeb2c89ccf3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3"
Nov 02 15:15:21.056480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.056461    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"
Nov 02 15:15:21.056565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.056492    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw" event={"ID":"b5075de1-a46e-4c77-9098-58f98918d782","Type":"ContainerDied","Data":"3d36c272262e8c40c317fbb9ce0c4b7cb19dc57b87b9725c7486de04dc3414c2"}
Nov 02 15:15:21.058765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.058742    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerStarted","Data":"b492890765efd294f0c6c0a725d3a27129f1570afa206ae4b31e2cdab2f79b41"}
Nov 02 15:15:21.068262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.068245    5041 scope.go:122] "RemoveContainer" containerID="f7774098fd0d90be614b181e63bf7d166aed6b752aa56f5fb09b2ba3f4b923b6"
Nov 02 15:15:21.083980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.083956    5041 scope.go:122] "RemoveContainer" containerID="1e786a60d920fe3159e0798713c3702ab811c143c0b9042381cd2b8fbf93077d"
Nov 02 15:15:21.139474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.139447    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"]
Nov 02 15:15:21.202344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.202283    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"]
Nov 02 15:15:21.212079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.212055    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893875-96xcw"]
Nov 02 15:15:21.220002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.219976    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"]
Nov 02 15:15:21.891700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.891665    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5075de1-a46e-4c77-9098-58f98918d782" path="/var/lib/kubelet/pods/b5075de1-a46e-4c77-9098-58f98918d782/volumes"
Nov 02 15:15:21.892154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:21.892139    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" path="/var/lib/kubelet/pods/e6d847d0-1640-433a-8b17-71ab99a033a3/volumes"
Nov 02 15:15:22.075751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.075717    5041 generic.go:356] "Generic (PLEG): container finished" podID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerID="b492890765efd294f0c6c0a725d3a27129f1570afa206ae4b31e2cdab2f79b41" exitCode=60
Nov 02 15:15:22.076119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.075794    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerDied","Data":"b492890765efd294f0c6c0a725d3a27129f1570afa206ae4b31e2cdab2f79b41"}
Nov 02 15:15:22.076119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.075863    5041 scope.go:122] "RemoveContainer" containerID="1fabcd3ae88dca14bc4a3998034e80411bb0ea87b03776086451d60634a2b3ed"
Nov 02 15:15:22.076330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.076312    5041 scope.go:122] "RemoveContainer" containerID="b492890765efd294f0c6c0a725d3a27129f1570afa206ae4b31e2cdab2f79b41"
Nov 02 15:15:22.076648 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:22.076628    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz_odh-ai-gateway-infra(d8a6b98a-106a-4aeb-a72b-b410a86e9e71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71"
Nov 02 15:15:22.138243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.138209    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"]
Nov 02 15:15:22.330246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.330215    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:22.332473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.332451    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:22.503335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.499785    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd\" (UniqueName: \"kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd\") pod \"90527bb2-3077-49bb-be87-9bffcd0c5caa\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") "
Nov 02 15:15:22.503335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.500019    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca\") pod \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") "
Nov 02 15:15:22.503335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.500139    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca\") pod \"90527bb2-3077-49bb-be87-9bffcd0c5caa\" (UID: \"90527bb2-3077-49bb-be87-9bffcd0c5caa\") "
Nov 02 15:15:22.503335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.500194    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg\" (UniqueName: \"kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg\") pod \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\" (UID: \"a1711e30-02c3-4008-8bf9-deeb2c89ccf3\") "
Nov 02 15:15:22.503335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.501343    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca" pod "a1711e30-02c3-4008-8bf9-deeb2c89ccf3" (UID: "a1711e30-02c3-4008-8bf9-deeb2c89ccf3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:22.503849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.503612    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca" pod "90527bb2-3077-49bb-be87-9bffcd0c5caa" (UID: "90527bb2-3077-49bb-be87-9bffcd0c5caa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:22.525388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.506330    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg" pod "a1711e30-02c3-4008-8bf9-deeb2c89ccf3" (UID: "a1711e30-02c3-4008-8bf9-deeb2c89ccf3"). InnerVolumeSpecName "kube-api-access-snvcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:22.525388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.508711    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd" pod "90527bb2-3077-49bb-be87-9bffcd0c5caa" (UID: "90527bb2-3077-49bb-be87-9bffcd0c5caa"). InnerVolumeSpecName "kube-api-access-cbnbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:22.602735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.602701    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbnbd\" (UniqueName: \"kubernetes.io/projected/90527bb2-3077-49bb-be87-9bffcd0c5caa-kube-api-access-cbnbd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:22.602735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.602731    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:22.602735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.602741    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90527bb2-3077-49bb-be87-9bffcd0c5caa-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:22.602962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:22.602749    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvcg\" (UniqueName: \"kubernetes.io/projected/a1711e30-02c3-4008-8bf9-deeb2c89ccf3-kube-api-access-snvcg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:23.090170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.090134    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4" event={"ID":"90527bb2-3077-49bb-be87-9bffcd0c5caa","Type":"ContainerDied","Data":"35fa89bb34b21f8557548df1623108c23f426eced735d8b421940796725aa3af"}
Nov 02 15:15:23.090170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.090156    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"
Nov 02 15:15:23.090623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.090201    5041 scope.go:122] "RemoveContainer" containerID="ea67e6fcf07f1febf4930a3137b23c7067abbddadc4de7d4746548d617a71f8b"
Nov 02 15:15:23.091987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.091962    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"
Nov 02 15:15:23.092113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.091963    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587" event={"ID":"a1711e30-02c3-4008-8bf9-deeb2c89ccf3","Type":"ContainerDied","Data":"d1799a3d232a753626e13ae12261becb1b348ecb63b14e2f7c68071fb1fadf7f"}
Nov 02 15:15:23.103395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.103376    5041 scope.go:122] "RemoveContainer" containerID="fc1faea123ed12e627efa302b40b5dd2878f8c7b2bab46bdfc7b12a4f4814e90"
Nov 02 15:15:23.161371 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.161328    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"]
Nov 02 15:15:23.171620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.171591    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893875-nb587"]
Nov 02 15:15:23.195062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.195037    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"]
Nov 02 15:15:23.203637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.203611    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893875-f22l4"]
Nov 02 15:15:23.268650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.268624    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:23.319303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.319272    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs\" (UniqueName: \"kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs\") pod \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") "
Nov 02 15:15:23.319709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.319681    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca\") pod \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\" (UID: \"d8a6b98a-106a-4aeb-a72b-b410a86e9e71\") "
Nov 02 15:15:23.320682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.320652    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca" pod "d8a6b98a-106a-4aeb-a72b-b410a86e9e71" (UID: "d8a6b98a-106a-4aeb-a72b-b410a86e9e71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:23.320896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.320872    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:23.322627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.322598    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs" pod "d8a6b98a-106a-4aeb-a72b-b410a86e9e71" (UID: "d8a6b98a-106a-4aeb-a72b-b410a86e9e71"). InnerVolumeSpecName "kube-api-access-kpljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:23.422182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.422153    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpljs\" (UniqueName: \"kubernetes.io/projected/d8a6b98a-106a-4aeb-a72b-b410a86e9e71-kube-api-access-kpljs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:15:23.890919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.890843    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" path="/var/lib/kubelet/pods/90527bb2-3077-49bb-be87-9bffcd0c5caa/volumes"
Nov 02 15:15:23.891284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:23.891270    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" path="/var/lib/kubelet/pods/a1711e30-02c3-4008-8bf9-deeb2c89ccf3/volumes"
Nov 02 15:15:24.140389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:24.134072    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz" event={"ID":"d8a6b98a-106a-4aeb-a72b-b410a86e9e71","Type":"ContainerDied","Data":"ac4ee80cf7e9043391858ce27816ceb94da2ae0cf30fda7d883723032a8b73e9"}
Nov 02 15:15:24.140389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:24.134156    5041 scope.go:122] "RemoveContainer" containerID="b492890765efd294f0c6c0a725d3a27129f1570afa206ae4b31e2cdab2f79b41"
Nov 02 15:15:24.140389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:24.139500    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"
Nov 02 15:15:24.217574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:24.217525    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"]
Nov 02 15:15:24.227288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:24.227256    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893875-r8hvz"]
Nov 02 15:15:25.890320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:25.890289    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" path="/var/lib/kubelet/pods/d8a6b98a-106a-4aeb-a72b-b410a86e9e71/volumes"
Nov 02 15:15:27.081611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:27.081582    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:15:30.036299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:30.036263    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"]
Nov 02 15:15:30.044908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:30.044880    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f8d8eb2-v792h"]
Nov 02 15:15:31.889950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:31.889917    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936b72c8-4d36-430c-a469-01d2d2f85120" path="/var/lib/kubelet/pods/936b72c8-4d36-430c-a469-01d2d2f85120/volumes"
Nov 02 15:15:36.769870 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:15:36.769805    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:15:42.036618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:42.036586    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"]
Nov 02 15:15:42.046619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:42.046583    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6742598b-574j4"]
Nov 02 15:15:43.037359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.037312    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"]
Nov 02 15:15:43.047477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.047448    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d90b7fe-n6mh5"]
Nov 02 15:15:43.053591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.053562    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"]
Nov 02 15:15:43.059327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.059300    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f401c0bd-n48p8"]
Nov 02 15:15:43.890794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.890762    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8853fb03-fa8a-4723-b811-0e9de3866d37" path="/var/lib/kubelet/pods/8853fb03-fa8a-4723-b811-0e9de3866d37/volumes"
Nov 02 15:15:43.891283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.891264    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c163f924-dc2f-4416-8d58-a5e0319dff6e" path="/var/lib/kubelet/pods/c163f924-dc2f-4416-8d58-a5e0319dff6e/volumes"
Nov 02 15:15:43.891739 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:43.891718    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110" path="/var/lib/kubelet/pods/f6b1aaa3-9bb4-4cec-ae31-e7491e5ba110/volumes"
Nov 02 15:15:44.885518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:44.885481    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:15:44.885937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:44.885569    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:45.530124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:45.530097    5041 generic.go:356] "Generic (PLEG): container finished" podID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerID="dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d" exitCode=0
Nov 02 15:15:45.530264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:45.530164    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" event={"ID":"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0","Type":"ContainerDied","Data":"dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"}
Nov 02 15:15:45.530643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:45.530627    5041 scope.go:122] "RemoveContainer" containerID="dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"
Nov 02 15:15:45.532321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:45.532291    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:15:45.532441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:45.532373    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:15:45.532441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:45.532416    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:15:46.546492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:46.546450    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" event={"ID":"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0","Type":"ContainerStarted","Data":"c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38"}
Nov 02 15:15:46.969415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:46.965977    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:15:46.969415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:46.966096    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 15:15:46.969415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:46.966167    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 15:15:47.957545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:47.957514    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 15:15:47.957925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:47.957563    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 15:15:48.776250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:48.776218    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:15:48.778695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:48.778661    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:15:48.958380 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:48.958333    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 15:15:48.958835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:48.958404    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 15:15:49.448372 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.448328    5041 scope.go:122] "RemoveContainer" containerID="b0f2ed061b51e72b4ea1efb4cbc6d66f0c5ad46fbd406024c38465cba25d3e2e"
Nov 02 15:15:49.460941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.460912    5041 scope.go:122] "RemoveContainer" containerID="d8d613f4b1eb60ede0ed5f12fa96df2f457e1da1314afcd38eed976cc15a19de"
Nov 02 15:15:49.471377 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.471343    5041 scope.go:122] "RemoveContainer" containerID="1f96fa398e791c7c7573f130915a5add48b109879417a01c5aa4051aa8727750"
Nov 02 15:15:49.481068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.481047    5041 scope.go:122] "RemoveContainer" containerID="6d74da1eff39cc1c4ef1d1f72a4c86679c3e03279b3fd7810710149a04cae0f2"
Nov 02 15:15:49.492069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.492049    5041 scope.go:122] "RemoveContainer" containerID="38cfff588cedc0f5b6f9cafcc4342366f761618f8c9a86adfc364694ae50ab79"
Nov 02 15:15:49.502663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.502633    5041 scope.go:122] "RemoveContainer" containerID="d8c80bd88e63fb9c32b8dd9e978c597f47d3ae09b8e4d7cd5cb1cc522ae1744d"
Nov 02 15:15:49.537685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.537592    5041 scope.go:122] "RemoveContainer" containerID="c0533ce0dd59feca9be265d0d35439d8bc9032b2341bd9db0cb9b85f460c0d77"
Nov 02 15:15:49.966388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.962516    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 15:15:49.966388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:49.962607    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 15:15:50.957516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:50.957479    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 15:15:50.957773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:50.957534    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 15:15:51.958047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:51.958012    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 15:15:51.958485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:51.958062    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 15:15:52.542500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.542455    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"]
Nov 02 15:15:52.543707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543679    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.543707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543705    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543736    5041 cpu_manager.go:414] "containerMap: removing container" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543745    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543762    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543769    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543783    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543791    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543810    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543817    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543830    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543838    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543852    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543859    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543866    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543870    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543889    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543896    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543938    5041 cpu_manager.go:414] "containerMap: removing container" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543945    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.543957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543964    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.543971    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544140    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544157    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544170    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544179    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544189    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544200    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544211    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544219    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544229    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544240    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544250    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5075de1-a46e-4c77-9098-58f98918d782" containerName="cleanup"
Nov 02 15:15:52.544725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.544261    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.550570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.550545    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:52.567281 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.567255    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-cce791-dockercfg-pmz77\""
Nov 02 15:15:52.567587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.567534    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-cce791\""
Nov 02 15:15:52.572826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.572802    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"]
Nov 02 15:15:52.653386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.652754    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:52.653386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.653061    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fh9z\" (UniqueName: \"kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z\") pod \"payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:52.858442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.857777    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fh9z\" (UniqueName: \"kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z\") pod \"payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:52.858442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.857961    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:52.859206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.859147    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.926383    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"]
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927633    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927657    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927712    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927720    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927758    5041 cpu_manager.go:414] "containerMap: removing container" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927767    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927792    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927801    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1711e30-02c3-4008-8bf9-deeb2c89ccf3" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.927977    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d847d0-1640-433a-8b17-71ab99a033a3" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.928002    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a6b98a-106a-4aeb-a72b-b410a86e9e71" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.928013    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="90527bb2-3077-49bb-be87-9bffcd0c5caa" containerName="cleanup"
Nov 02 15:15:52.940386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:52.937270    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.020680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.000921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fh9z\" (UniqueName: \"kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z\") pod \"payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:53.022014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.021974    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g958n\" (UniqueName: \"kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n\") pod \"payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.022238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.022214    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.049954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.049919    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"]
Nov 02 15:15:53.055688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.055660    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:15:53.065477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.065447    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:15:53.067529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.067504    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:15:53.079715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.078154    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"]
Nov 02 15:15:53.086915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.086890    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab3e53-rzcpb"]
Nov 02 15:15:53.123904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.123837    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.124107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.124090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g958n\" (UniqueName: \"kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n\") pod \"payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.124620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.124600    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.147172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.147132    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g958n\" (UniqueName: \"kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n\") pod \"payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.168741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.168715    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:53.364871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.364839    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:53.483759 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:53.483716    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6539ec_29c7_41d1_ba11_0d1a920530ae.slice/crio-2f3da8b09941a5300dd6916de544af632288209323b3e728ecbd34f0572b47a5 WatchSource:0}: Error finding container 2f3da8b09941a5300dd6916de544af632288209323b3e728ecbd34f0572b47a5: Status 404 returned error can't find the container with id 2f3da8b09941a5300dd6916de544af632288209323b3e728ecbd34f0572b47a5
Nov 02 15:15:53.487356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.487290    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"]
Nov 02 15:15:53.675560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.675524    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"]
Nov 02 15:15:53.675779 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:53.675758    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a1a0fb_bea2_4061_b32c_67ed9afeb526.slice/crio-a3be20514ed31e8c3813e6bf9e59dba2c831212f6c34a97966f9ba171e1fb6cf WatchSource:0}: Error finding container a3be20514ed31e8c3813e6bf9e59dba2c831212f6c34a97966f9ba171e1fb6cf: Status 404 returned error can't find the container with id a3be20514ed31e8c3813e6bf9e59dba2c831212f6c34a97966f9ba171e1fb6cf
Nov 02 15:15:53.889933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.889889    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8b124f-bc48-4e66-83b9-d15bd525d842" path="/var/lib/kubelet/pods/5e8b124f-bc48-4e66-83b9-d15bd525d842/volumes"
Nov 02 15:15:53.958618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.958533    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" event={"ID":"fd6539ec-29c7-41d1-ba11-0d1a920530ae","Type":"ContainerStarted","Data":"dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483"}
Nov 02 15:15:53.958618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.958598    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" event={"ID":"fd6539ec-29c7-41d1-ba11-0d1a920530ae","Type":"ContainerStarted","Data":"2f3da8b09941a5300dd6916de544af632288209323b3e728ecbd34f0572b47a5"}
Nov 02 15:15:53.958850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.958635    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:15:53.960425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.960395    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" event={"ID":"50a1a0fb-bea2-4061-b32c-67ed9afeb526","Type":"ContainerStarted","Data":"a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad"}
Nov 02 15:15:53.960555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.960451    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" event={"ID":"50a1a0fb-bea2-4061-b32c-67ed9afeb526","Type":"ContainerStarted","Data":"a3be20514ed31e8c3813e6bf9e59dba2c831212f6c34a97966f9ba171e1fb6cf"}
Nov 02 15:15:53.960555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:53.960480    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:15:54.000670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.000622    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" podStartSLOduration=2.000608243 podStartE2EDuration="2.000608243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:53.994299394 +0000 UTC m=+1206.252368845" watchObservedRunningTime="2026-11-02 15:15:54.000608243 +0000 UTC m=+1206.258677694"
Nov 02 15:15:54.033679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.033632    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" podStartSLOduration=2.033618407 podStartE2EDuration="2.033618407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:54.033029567 +0000 UTC m=+1206.291099017" watchObservedRunningTime="2026-11-02 15:15:54.033618407 +0000 UTC m=+1206.291687856"
Nov 02 15:15:54.549742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.549706    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"]
Nov 02 15:15:54.554329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.554304    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:54.569899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.569880    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d7df16e\""
Nov 02 15:15:54.570111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.570092    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3d7df16e-dockercfg-zwlsh\""
Nov 02 15:15:54.576346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.576323    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"]
Nov 02 15:15:54.652647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.652583    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:54.653115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.653089    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:54.794613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.794575    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"]
Nov 02 15:15:54.850435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.844742    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:54.850435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.844954    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:54.850435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.846070    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:54.989433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.935593    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:54.991153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:54.988718    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"]
Nov 02 15:15:55.026672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.022110    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:55.061404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.061373    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.061733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.061636    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q4r\" (UniqueName: \"kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.168018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.167848    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.168536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.168501    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q4r\" (UniqueName: \"kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.168737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.168708    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.170024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.170001    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:55.190643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.190620    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q4r\" (UniqueName: \"kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.347271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.347231    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:55.499619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.499577    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"]
Nov 02 15:15:55.504613 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:55.504577    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f200b8e_cc5e_4428_a36a_254ec0d1b8d7.slice/crio-9e420f6ba3388ea8d17a4227060445719bcef893dc4fc1dd796afa74f63a9b66 WatchSource:0}: Error finding container 9e420f6ba3388ea8d17a4227060445719bcef893dc4fc1dd796afa74f63a9b66: Status 404 returned error can't find the container with id 9e420f6ba3388ea8d17a4227060445719bcef893dc4fc1dd796afa74f63a9b66
Nov 02 15:15:55.660736 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:15:55.660707    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75fe8c9_9fc6_420d_9ce2_857794e80e16.slice/crio-6526a4226ca30487e5b3bfc7b1d9c77d3fc4271c0e430d07c6c61389e353ba99 WatchSource:0}: Error finding container 6526a4226ca30487e5b3bfc7b1d9c77d3fc4271c0e430d07c6c61389e353ba99: Status 404 returned error can't find the container with id 6526a4226ca30487e5b3bfc7b1d9c77d3fc4271c0e430d07c6c61389e353ba99
Nov 02 15:15:55.662541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:55.662513    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"]
Nov 02 15:15:56.052159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.052073    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" event={"ID":"a75fe8c9-9fc6-420d-9ce2-857794e80e16","Type":"ContainerStarted","Data":"0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047"}
Nov 02 15:15:56.052159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.052136    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" event={"ID":"a75fe8c9-9fc6-420d-9ce2-857794e80e16","Type":"ContainerStarted","Data":"6526a4226ca30487e5b3bfc7b1d9c77d3fc4271c0e430d07c6c61389e353ba99"}
Nov 02 15:15:56.052347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.052164    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:15:56.053931 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.053900    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" event={"ID":"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7","Type":"ContainerStarted","Data":"126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da"}
Nov 02 15:15:56.054080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.053950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" event={"ID":"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7","Type":"ContainerStarted","Data":"9e420f6ba3388ea8d17a4227060445719bcef893dc4fc1dd796afa74f63a9b66"}
Nov 02 15:15:56.054080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.053987    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:15:56.097687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.097642    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" podStartSLOduration=2.097629596 podStartE2EDuration="2.097629596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:56.090882782 +0000 UTC m=+1208.348952242" watchObservedRunningTime="2026-11-02 15:15:56.097629596 +0000 UTC m=+1208.355699039"
Nov 02 15:15:56.133409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:56.133353    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" podStartSLOduration=2.133342236 podStartE2EDuration="2.133342236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:56.127961858 +0000 UTC m=+1208.386031307" watchObservedRunningTime="2026-11-02 15:15:56.133342236 +0000 UTC m=+1208.391411685"
Nov 02 15:15:57.033606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:57.033575    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"]
Nov 02 15:15:57.043003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:57.042974    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db3d36a-99vl9"]
Nov 02 15:15:57.890748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:15:57.890715    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03079dc-4006-4658-a57d-cfe73dc2e567" path="/var/lib/kubelet/pods/d03079dc-4006-4658-a57d-cfe73dc2e567/volumes"
Nov 02 15:16:02.034507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:02.034433    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"]
Nov 02 15:16:02.042982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:02.042956    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2697c2-mbtc5"]
Nov 02 15:16:03.114031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:03.113996    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"]
Nov 02 15:16:03.126121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:03.126090    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa5874c-rtlj6"]
Nov 02 15:16:03.890290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:03.890259    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7157c814-23ee-4f1d-b4e6-82723bdbd3a5" path="/var/lib/kubelet/pods/7157c814-23ee-4f1d-b4e6-82723bdbd3a5/volumes"
Nov 02 15:16:03.890754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:03.890740    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a7a3d4-889c-4316-afc2-ff3ee94e60bc" path="/var/lib/kubelet/pods/f3a7a3d4-889c-4316-afc2-ff3ee94e60bc/volumes"
Nov 02 15:16:05.044303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:05.044271    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:16:05.045767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:05.045737    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:16:06.770514 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:06.770477    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:16:06.854318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:06.854289    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"]
Nov 02 15:16:06.858676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:06.858653    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:06.874536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:06.874510    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-246137\""
Nov 02 15:16:06.874674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:06.874575    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-246137-dockercfg-xdn7z\""
Nov 02 15:16:06.893905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:06.893877    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"]
Nov 02 15:16:07.076501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.058681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bl6\" (UniqueName: \"kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6\") pod \"payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.076501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.058971    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.094387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.090922    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"]
Nov 02 15:16:07.194385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.184993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bl6\" (UniqueName: \"kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6\") pod \"payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.194385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.185201    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.194385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.186182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.354651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.307431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bl6\" (UniqueName: \"kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6\") pod \"payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.354817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.354649    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:16:07.354817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.354739    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:16:07.357458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.355855    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.366426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.365585    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"]
Nov 02 15:16:07.477084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.477055    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:07.523901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.523844    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkq4\" (UniqueName: \"kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4\") pod \"payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.524231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.524128    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.746924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.746891    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.749160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.747754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.749160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.748849    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkq4\" (UniqueName: \"kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4\") pod \"payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.770212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.770182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkq4\" (UniqueName: \"kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4\") pod \"payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.982846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.982815    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:07.991557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:07.991531    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"]
Nov 02 15:16:07.992520 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:16:07.992498    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e84e50_c665_4c63_9744_106f9c0dc549.slice/crio-d0b936303fefd38db88697b2ca745fbb1d456901aba82961bbd78b75da18f5f6 WatchSource:0}: Error finding container d0b936303fefd38db88697b2ca745fbb1d456901aba82961bbd78b75da18f5f6: Status 404 returned error can't find the container with id d0b936303fefd38db88697b2ca745fbb1d456901aba82961bbd78b75da18f5f6
Nov 02 15:16:08.479385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:08.471460    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"]
Nov 02 15:16:08.492451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:08.492416    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" event={"ID":"bd42e35e-24c6-4644-a98a-a790e91652eb","Type":"ContainerStarted","Data":"2a8be6369681b63c0da02823d0df4a43285f1df124b803901f5d047d102f0dd2"}
Nov 02 15:16:08.494522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:08.494494    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" event={"ID":"60e84e50-c665-4c63-9744-106f9c0dc549","Type":"ContainerStarted","Data":"07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48"}
Nov 02 15:16:08.494654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:08.494541    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" event={"ID":"60e84e50-c665-4c63-9744-106f9c0dc549","Type":"ContainerStarted","Data":"d0b936303fefd38db88697b2ca745fbb1d456901aba82961bbd78b75da18f5f6"}
Nov 02 15:16:08.494654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:08.494588    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:08.555962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:08.555903    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" podStartSLOduration=2.555884838 podStartE2EDuration="2.555884838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:08.551866211 +0000 UTC m=+1220.809935662" watchObservedRunningTime="2026-11-02 15:16:08.555884838 +0000 UTC m=+1220.813954285"
Nov 02 15:16:09.505824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:09.505788    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" event={"ID":"bd42e35e-24c6-4644-a98a-a790e91652eb","Type":"ContainerStarted","Data":"158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121"}
Nov 02 15:16:09.506263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:09.505888    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:09.546613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:09.546557    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" podStartSLOduration=3.546544875 podStartE2EDuration="3.546544875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:09.540634877 +0000 UTC m=+1221.798704327" watchObservedRunningTime="2026-11-02 15:16:09.546544875 +0000 UTC m=+1221.804614324"
Nov 02 15:16:10.155580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.155541    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"]
Nov 02 15:16:10.164781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.164743    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.185390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.180325    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"]
Nov 02 15:16:10.198591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.198558    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-cce791-serving-cert\""
Nov 02 15:16:10.223450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.223419    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.223588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.223509    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.223685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.223661    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrt6s\" (UniqueName: \"kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.223840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.223821    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.347229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.331417    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.349321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.349284    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.363346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.359576    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrt6s\" (UniqueName: \"kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.363346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.359775    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.363346 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:10.360099    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-serving-cert: secret "maas-api-e2e-shared-b-w1-cce791-serving-cert" not found
Nov 02 15:16:10.363346 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:10.360180    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls podName:79b25021-d58b-46c7-95a0-5f13c7ceb327 nodeName:}" failed. No retries permitted until 2026-11-02 15:16:10.860160576 +0000 UTC m=+1223.118230005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls") pod "maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" (UID: "79b25021-d58b-46c7-95a0-5f13c7ceb327") : secret "maas-api-e2e-shared-b-w1-cce791-serving-cert" not found
Nov 02 15:16:10.364734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.336232    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.403166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.403129    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.423442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.411499    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrt6s\" (UniqueName: \"kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.536979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.536944    5041 generic.go:356] "Generic (PLEG): container finished" podID="fe12de84-1c2d-492b-88ac-c3ca2c327b03" containerID="527153f63042b9fc6cfe45c5b051ac4e201ea4352d658f0302838950349102a8" exitCode=0
Nov 02 15:16:10.537438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.537015    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7m7n-qffpv" event={"ID":"fe12de84-1c2d-492b-88ac-c3ca2c327b03","Type":"ContainerDied","Data":"527153f63042b9fc6cfe45c5b051ac4e201ea4352d658f0302838950349102a8"}
Nov 02 15:16:10.874514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.874425    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:10.876851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:10.876829    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:11.089975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.089942    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:11.448713 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:16:11.448678    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b25021_d58b_46c7_95a0_5f13c7ceb327.slice/crio-bdfcb4d0e45062bd3d8d9507328d2f97d25e7acb4e7788c6bd150f5d54102b3f WatchSource:0}: Error finding container bdfcb4d0e45062bd3d8d9507328d2f97d25e7acb4e7788c6bd150f5d54102b3f: Status 404 returned error can't find the container with id bdfcb4d0e45062bd3d8d9507328d2f97d25e7acb4e7788c6bd150f5d54102b3f
Nov 02 15:16:11.449942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.449922    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"]
Nov 02 15:16:11.778459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.776790    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" event={"ID":"79b25021-d58b-46c7-95a0-5f13c7ceb327","Type":"ContainerStarted","Data":"bdfcb4d0e45062bd3d8d9507328d2f97d25e7acb4e7788c6bd150f5d54102b3f"}
Nov 02 15:16:11.792088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.792049    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"]
Nov 02 15:16:11.806626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.798436    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:11.827148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.818728    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c7afd1a6-dockercfg-rfkgg\""
Nov 02 15:16:11.827148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.818817    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c7afd1a6\""
Nov 02 15:16:11.838324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.838289    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"]
Nov 02 15:16:11.863381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.855165    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"]
Nov 02 15:16:11.868382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:11.867559    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.033527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.016820    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h45v\" (UniqueName: \"kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v\") pod \"payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.033527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.016981    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") " pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.033527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.017017    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.033527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.017099    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlphz\" (UniqueName: \"kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz\") pod \"payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") " pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.129911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.129861    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") " pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.130202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.130171    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.130431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.130414    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlphz\" (UniqueName: \"kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz\") pod \"payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") " pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.130942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.130923    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h45v\" (UniqueName: \"kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v\") pod \"payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.159868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.133179    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") " pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.163620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.133917    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.280314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.279211    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h45v\" (UniqueName: \"kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v\") pod \"payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.334827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.318497    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"]
Nov 02 15:16:12.334827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.319616    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlphz\" (UniqueName: \"kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz\") pod \"payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") " pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.444869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.444835    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:12.451007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.450976    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:12.588925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.588648    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:16:12.646700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.646642    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"]
Nov 02 15:16:12.647921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.647893    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fe12de84-1c2d-492b-88ac-c3ca2c327b03" containerName="insights-gathering"
Nov 02 15:16:12.647921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.647919    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe12de84-1c2d-492b-88ac-c3ca2c327b03" containerName="insights-gathering"
Nov 02 15:16:12.648178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.648147    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe12de84-1c2d-492b-88ac-c3ca2c327b03" containerName="insights-gathering"
Nov 02 15:16:12.658358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.657090    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.672059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.672030    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"]
Nov 02 15:16:12.697666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.697635    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle\") pod \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") "
Nov 02 15:16:12.697832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.697700    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path\") pod \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") "
Nov 02 15:16:12.697936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.697908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle" pod "fe12de84-1c2d-492b-88ac-c3ca2c327b03" (UID: "fe12de84-1c2d-492b-88ac-c3ca2c327b03"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:12.698112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.698076    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz\" (UniqueName: \"kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz\") pod \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") "
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.698845    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle\") pod \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\" (UID: \"fe12de84-1c2d-492b-88ac-c3ca2c327b03\") "
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.699233    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgp2\" (UniqueName: \"kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.699318    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.700559    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle" pod "fe12de84-1c2d-492b-88ac-c3ca2c327b03" (UID: "fe12de84-1c2d-492b-88ac-c3ca2c327b03"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.700606    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path" pod "fe12de84-1c2d-492b-88ac-c3ca2c327b03" (UID: "fe12de84-1c2d-492b-88ac-c3ca2c327b03"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.700868    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.700899    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.701225    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-trusted-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.701243    5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fe12de84-1c2d-492b-88ac-c3ca2c327b03-archives-path\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:16:12.701568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.701257    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe12de84-1c2d-492b-88ac-c3ca2c327b03-service-ca-bundle\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:16:12.708757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.703840    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz" pod "fe12de84-1c2d-492b-88ac-c3ca2c327b03" (UID: "fe12de84-1c2d-492b-88ac-c3ca2c327b03"). InnerVolumeSpecName "kube-api-access-7vvwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:12.708757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.704185    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d7df16e-serving-cert\""
Nov 02 15:16:12.770507 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:16:12.770476    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c8f6d6_8f77_4b25_a3ed_2b50602b3719.slice/crio-89a9485647e777ef85b3ebd4b57b52ca554ba37bd0267308fe48df4294480e9b WatchSource:0}: Error finding container 89a9485647e777ef85b3ebd4b57b52ca554ba37bd0267308fe48df4294480e9b: Status 404 returned error can't find the container with id 89a9485647e777ef85b3ebd4b57b52ca554ba37bd0267308fe48df4294480e9b
Nov 02 15:16:12.770823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.770798    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"]
Nov 02 15:16:12.793878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.793819    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5" event={"ID":"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719","Type":"ContainerStarted","Data":"89a9485647e777ef85b3ebd4b57b52ca554ba37bd0267308fe48df4294480e9b"}
Nov 02 15:16:12.798021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.796109    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7m7n-qffpv"
Nov 02 15:16:12.798021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.797901    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7m7n-qffpv" event={"ID":"fe12de84-1c2d-492b-88ac-c3ca2c327b03","Type":"ContainerDied","Data":"61c7e3a7fd75dac459eda247b8a83bfd215258a79271fe955f6e34d4329dd81a"}
Nov 02 15:16:12.798021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.797978    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61c7e3a7fd75dac459eda247b8a83bfd215258a79271fe955f6e34d4329dd81a"
Nov 02 15:16:12.807410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.807209    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgp2\" (UniqueName: \"kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.807410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.807299    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.807546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.807495    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.807546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.807530    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.808812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.808787    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvwz\" (UniqueName: \"kubernetes.io/projected/fe12de84-1c2d-492b-88ac-c3ca2c327b03-kube-api-access-7vvwz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:16:12.809805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.809777    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.810063 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:12.810024    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-serving-cert: secret "maas-api-e2e-mt-3d7df16e-serving-cert" not found
Nov 02 15:16:12.810153 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:12.810097    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls podName:b26f0bf2-f8f7-416d-ba83-2c4b58c742b2 nodeName:}" failed. No retries permitted until 2026-11-02 15:16:13.310078244 +0000 UTC m=+1225.568147686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls") pod "maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" (UID: "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2") : secret "maas-api-e2e-mt-3d7df16e-serving-cert" not found
Nov 02 15:16:12.812191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.812121    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" event={"ID":"79b25021-d58b-46c7-95a0-5f13c7ceb327","Type":"ContainerStarted","Data":"a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600"}
Nov 02 15:16:12.839504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.818880    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.865431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.843949    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:12.883245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.883214    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgp2\" (UniqueName: \"kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:12.986615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.922989    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"]
Nov 02 15:16:12.986902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:12.942681    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" podStartSLOduration=2.115019548 podStartE2EDuration="2.942660385s" totalImagesPullingTime="827.640837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:12.885274748 +0000 UTC m=+1225.143344198" watchObservedRunningTime="2026-11-02 15:16:12.942660385 +0000 UTC m=+1225.200729836"
Nov 02 15:16:13.350801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.350773    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:13.353096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.353075    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:13.586538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.586465    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:13.826387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.826330    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5" event={"ID":"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719","Type":"ContainerStarted","Data":"c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2"}
Nov 02 15:16:13.826939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.826917    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:13.828526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.828502    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb" event={"ID":"ab3f400f-be12-41bc-86a6-f0a1723ea179","Type":"ContainerStarted","Data":"2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64"}
Nov 02 15:16:13.828624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.828550    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb" event={"ID":"ab3f400f-be12-41bc-86a6-f0a1723ea179","Type":"ContainerStarted","Data":"a55a6ec5ceb9225f9377fce98533591e55a88a3ce056115014b2ffe98b794896"}
Nov 02 15:16:13.828624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.828599    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:13.863983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.863907    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5" podStartSLOduration=2.86389417 podStartE2EDuration="2.86389417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:13.862954059 +0000 UTC m=+1226.121023509" watchObservedRunningTime="2026-11-02 15:16:13.86389417 +0000 UTC m=+1226.121963619"
Nov 02 15:16:13.899976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.899933    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb" podStartSLOduration=2.899922124 podStartE2EDuration="2.899922124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:13.898462877 +0000 UTC m=+1226.156532325" watchObservedRunningTime="2026-11-02 15:16:13.899922124 +0000 UTC m=+1226.157991573"
Nov 02 15:16:13.911562 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:16:13.911529    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26f0bf2_f8f7_416d_ba83_2c4b58c742b2.slice/crio-7bb691c5aee37d8b9cc5b829213f964b4acd0ef423d721ca07af108f4d8d107c WatchSource:0}: Error finding container 7bb691c5aee37d8b9cc5b829213f964b4acd0ef423d721ca07af108f4d8d107c: Status 404 returned error can't find the container with id 7bb691c5aee37d8b9cc5b829213f964b4acd0ef423d721ca07af108f4d8d107c
Nov 02 15:16:13.914714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:13.914688    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"]
Nov 02 15:16:14.953558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:14.952743    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerStarted","Data":"661a0699143542bc30af24d4c46eafd1855519bcfccf1cc9b87cd669fe4ebf92"}
Nov 02 15:16:14.953558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:14.952809    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerStarted","Data":"7bb691c5aee37d8b9cc5b829213f964b4acd0ef423d721ca07af108f4d8d107c"}
Nov 02 15:16:14.963667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:14.963618    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:15.001444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:15.001391    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podStartSLOduration=2.6136033850000002 podStartE2EDuration="3.00135792s" totalImagesPullingTime="387.754535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:14.997429294 +0000 UTC m=+1227.255498740" watchObservedRunningTime="2026-11-02 15:16:15.00135792 +0000 UTC m=+1227.259427400"
Nov 02 15:16:16.036599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:16.036566    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"]
Nov 02 15:16:16.046093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:16.046062    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea35dc8-zn72l"]
Nov 02 15:16:17.892056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:17.892021    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9011bd-2757-48d8-87dc-d37e650a17ef" path="/var/lib/kubelet/pods/5e9011bd-2757-48d8-87dc-d37e650a17ef/volumes"
Nov 02 15:16:18.042937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:18.042899    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"]
Nov 02 15:16:18.048681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:18.048654    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da75a42e-xskvz"]
Nov 02 15:16:19.509029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:19.508994    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:16:19.890833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:19.890755    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3f883e-4e20-4f33-bd8c-3cef098d0a84" path="/var/lib/kubelet/pods/3b3f883e-4e20-4f33-bd8c-3cef098d0a84/volumes"
Nov 02 15:16:19.957859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:19.957833    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:16:20.541316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:20.541288    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:16:21.978717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:21.978675    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.204:8443/health\": dial tcp 10.132.0.204:8443: connect: connection refused"
Nov 02 15:16:22.051830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:22.051792    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"]
Nov 02 15:16:22.057632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:22.057602    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80a3e2a-ztj48"]
Nov 02 15:16:23.891170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:23.891136    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4253f7d7-59f3-462a-884e-1a91303fd2f1" path="/var/lib/kubelet/pods/4253f7d7-59f3-462a-884e-1a91303fd2f1/volumes"
Nov 02 15:16:24.993799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:24.992554    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:16:25.083613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:25.083570    5041 generic.go:356] "Generic (PLEG): container finished" podID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerID="661a0699143542bc30af24d4c46eafd1855519bcfccf1cc9b87cd669fe4ebf92" exitCode=1
Nov 02 15:16:25.083865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:25.083848    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerDied","Data":"661a0699143542bc30af24d4c46eafd1855519bcfccf1cc9b87cd669fe4ebf92"}
Nov 02 15:16:25.084311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:25.084288    5041 scope.go:122] "RemoveContainer" containerID="661a0699143542bc30af24d4c46eafd1855519bcfccf1cc9b87cd669fe4ebf92"
Nov 02 15:16:25.963741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:25.963704    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:16:26.096332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:26.096300    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerStarted","Data":"7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584"}
Nov 02 15:16:26.096761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:26.096538    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:28.018520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.018486    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"]
Nov 02 15:16:28.022965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.022945    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.044520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.044495    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-246137-serving-cert\""
Nov 02 15:16:28.051059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.051035    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"]
Nov 02 15:16:28.054265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.054234    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttr2m\" (UniqueName: \"kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.054573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.054553    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.054790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.054773    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.054974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.054960    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.156970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.156933    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttr2m\" (UniqueName: \"kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.157164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.157036    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.157234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.157160    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.157290 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:28.157277    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-serving-cert: secret "maas-api-e2e-shared-a-w5-246137-serving-cert" not found
Nov 02 15:16:28.157345 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:28.157331    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls podName:3d0cce51-33f1-42ce-97bc-1a9829c2322e nodeName:}" failed. No retries permitted until 2026-11-02 15:16:28.657317846 +0000 UTC m=+1240.915387274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls") pod "maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" (UID: "3d0cce51-33f1-42ce-97bc-1a9829c2322e") : secret "maas-api-e2e-shared-a-w5-246137-serving-cert" not found
Nov 02 15:16:28.157491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.157281    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.157902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.157879    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.159521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.159502    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.281355 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.281273    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttr2m\" (UniqueName: \"kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.683105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.683073    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.685690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.685655    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:28.946014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:28.945947    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:29.264983 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:16:29.264952    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0cce51_33f1_42ce_97bc_1a9829c2322e.slice/crio-1b1f4fd7f44aa92c794a61e012ba78e1bad097235bbdd39c8bcd87d7c674393c WatchSource:0}: Error finding container 1b1f4fd7f44aa92c794a61e012ba78e1bad097235bbdd39c8bcd87d7c674393c: Status 404 returned error can't find the container with id 1b1f4fd7f44aa92c794a61e012ba78e1bad097235bbdd39c8bcd87d7c674393c
Nov 02 15:16:29.267178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:29.267147    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"]
Nov 02 15:16:30.033944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:30.033869    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"]
Nov 02 15:16:30.042990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:30.042954    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7d7a50e-b2kpb"]
Nov 02 15:16:30.296437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:30.288612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" event={"ID":"3d0cce51-33f1-42ce-97bc-1a9829c2322e","Type":"ContainerStarted","Data":"29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91"}
Nov 02 15:16:30.296437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:30.288664    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" event={"ID":"3d0cce51-33f1-42ce-97bc-1a9829c2322e","Type":"ContainerStarted","Data":"1b1f4fd7f44aa92c794a61e012ba78e1bad097235bbdd39c8bcd87d7c674393c"}
Nov 02 15:16:30.296437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:30.289761    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:30.344517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:30.344464    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" podStartSLOduration=2.909684812 podStartE2EDuration="3.344450065s" totalImagesPullingTime="434.765253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:30.338963901 +0000 UTC m=+1242.597033350" watchObservedRunningTime="2026-11-02 15:16:30.344450065 +0000 UTC m=+1242.602519514"
Nov 02 15:16:31.947243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:31.947197    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ec1ef6-d87b-4a4a-af8a-1b9b26aff134" path="/var/lib/kubelet/pods/48ec1ef6-d87b-4a4a-af8a-1b9b26aff134/volumes"
Nov 02 15:16:32.038538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:32.038502    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"]
Nov 02 15:16:32.044325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:32.044296    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f3801e-ktztk"]
Nov 02 15:16:32.106762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:32.106724    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.204:8443/health\": dial tcp 10.132.0.204:8443: connect: connection refused"
Nov 02 15:16:33.890843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:33.890798    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addee6de-7eb8-4939-a5f2-966830e970f8" path="/var/lib/kubelet/pods/addee6de-7eb8-4939-a5f2-966830e970f8/volumes"
Nov 02 15:16:34.935447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.935412    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"]
Nov 02 15:16:34.940122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.940102    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:34.964486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.964449    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c7afd1a6-serving-cert\""
Nov 02 15:16:34.969972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.969944    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"]
Nov 02 15:16:34.995244 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.995211    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgq6\" (UniqueName: \"kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:34.995407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.995382    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:34.995731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.995698    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:34.996068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:34.996047    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.098065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.098023    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.098195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.098153    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.098359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.098343    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.098515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.098495    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgq6\" (UniqueName: \"kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.098744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.098723    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.100764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.100742    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.101001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.100978    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.121057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.121030    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgq6\" (UniqueName: \"kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6\") pod \"maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.257940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.257831    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:35.635855 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:16:35.635824    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4d69c3_4795_41b3_a81c_4280f002f10d.slice/crio-300b43541fb7844142b0d0b02310ffd4b72eb2ec46d4da550ca6db772f5f27cc WatchSource:0}: Error finding container 300b43541fb7844142b0d0b02310ffd4b72eb2ec46d4da550ca6db772f5f27cc: Status 404 returned error can't find the container with id 300b43541fb7844142b0d0b02310ffd4b72eb2ec46d4da550ca6db772f5f27cc
Nov 02 15:16:35.636840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:35.636810    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"]
Nov 02 15:16:36.371806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.371761    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc" event={"ID":"7b4d69c3-4795-41b3-a81c-4280f002f10d","Type":"ContainerStarted","Data":"72a587b24c98a1ae334a578650ed807ba7726c17f28b2975700d60a8d36d94b4"}
Nov 02 15:16:36.372224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.371826    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc" event={"ID":"7b4d69c3-4795-41b3-a81c-4280f002f10d","Type":"ContainerStarted","Data":"300b43541fb7844142b0d0b02310ffd4b72eb2ec46d4da550ca6db772f5f27cc"}
Nov 02 15:16:36.372224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.371852    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:36.373989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.373961    5041 generic.go:356] "Generic (PLEG): container finished" podID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerID="7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584" exitCode=1
Nov 02 15:16:36.374115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.374041    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerDied","Data":"7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584"}
Nov 02 15:16:36.374115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.374103    5041 scope.go:122] "RemoveContainer" containerID="661a0699143542bc30af24d4c46eafd1855519bcfccf1cc9b87cd669fe4ebf92"
Nov 02 15:16:36.374532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.374508    5041 scope.go:122] "RemoveContainer" containerID="7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584"
Nov 02 15:16:36.374991 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:36.374898    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm_odh-ai-gateway-infra(b26f0bf2-f8f7-416d-ba83-2c4b58c742b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"
Nov 02 15:16:36.452090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:36.452027    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc" podStartSLOduration=2.186723409 podStartE2EDuration="2.45201017s" totalImagesPullingTime="265.286761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:36.445017505 +0000 UTC m=+1248.703086960" watchObservedRunningTime="2026-11-02 15:16:36.45201017 +0000 UTC m=+1248.710079621"
Nov 02 15:16:36.771629 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:36.771586    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:16:37.043789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.043696    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"]
Nov 02 15:16:37.049953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.049924    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"]
Nov 02 15:16:37.055338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.055312    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e006390-5rrdx"]
Nov 02 15:16:37.060129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.060104    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822e0586-vwnkm"]
Nov 02 15:16:37.314167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.314101    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:16:37.892651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.892621    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144923bb-260c-4905-9489-45062b69a29f" path="/var/lib/kubelet/pods/144923bb-260c-4905-9489-45062b69a29f/volumes"
Nov 02 15:16:37.893092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:37.893077    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962bfa09-6008-4180-85f8-3676289e6f9e" path="/var/lib/kubelet/pods/962bfa09-6008-4180-85f8-3676289e6f9e/volumes"
Nov 02 15:16:39.031883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:39.031850    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"]
Nov 02 15:16:39.040934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:39.040902    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0d8a77-4fhv8"]
Nov 02 15:16:39.891178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:39.891143    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d068689e-1232-43de-b883-baa1fa9f404c" path="/var/lib/kubelet/pods/d068689e-1232-43de-b883-baa1fa9f404c/volumes"
Nov 02 15:16:42.398002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:42.397970    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:16:43.586755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:43.586707    5041 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:43.587997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:43.587970    5041 scope.go:122] "RemoveContainer" containerID="7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584"
Nov 02 15:16:43.589281 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:43.589246    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm_odh-ai-gateway-infra(b26f0bf2-f8f7-416d-ba83-2c4b58c742b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"
Nov 02 15:16:44.032546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:44.032512    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"]
Nov 02 15:16:44.038069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:44.038043    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0e0184-mtwt4"]
Nov 02 15:16:45.880709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:45.880674    5041 scope.go:122] "RemoveContainer" containerID="7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584"
Nov 02 15:16:45.892285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:45.892251    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fc208a-fb36-4420-be1a-dbd040b79a99" path="/var/lib/kubelet/pods/a9fc208a-fb36-4420-be1a-dbd040b79a99/volumes"
Nov 02 15:16:46.621416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:46.621300    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerStarted","Data":"cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"}
Nov 02 15:16:46.621564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:46.621538    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:16:49.680486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:49.680457    5041 scope.go:122] "RemoveContainer" containerID="6aeff1068468130f639d125af6f9073f32c110d4b3ff17fd01cc920478a307e9"
Nov 02 15:16:49.692533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:49.692507    5041 scope.go:122] "RemoveContainer" containerID="67c70e56c9761f76b036a203caca5dd4f107168e022334ee54a48fe085ee21fc"
Nov 02 15:16:49.703478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:49.703446    5041 scope.go:122] "RemoveContainer" containerID="8b5d3fd2c4c7b08b292ced3842482f5c8be8fb54cf2fd8557fb9094c6117a50b"
Nov 02 15:16:49.714196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:49.714176    5041 scope.go:122] "RemoveContainer" containerID="195fdd62627f1f9d8b9ffc6c2979702725a83f7e41086620f2c3a33705ebbed5"
Nov 02 15:16:49.724194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:49.724171    5041 scope.go:122] "RemoveContainer" containerID="730b0d20e7d4df35082e175322dd39f4e0c13e1983f2c2f72fe2b13d8e46f3e3"
Nov 02 15:16:51.118389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:51.102546    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"]
Nov 02 15:16:51.191882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:51.191819    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"]
Nov 02 15:16:51.203516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:51.203486    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46338096-vhq9w"]
Nov 02 15:16:51.208685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:51.208653    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-670b724c-8jzdb"]
Nov 02 15:16:51.913054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:51.913014    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e48a660-18a0-42a5-aebd-bc8826719e61" path="/var/lib/kubelet/pods/3e48a660-18a0-42a5-aebd-bc8826719e61/volumes"
Nov 02 15:16:51.914072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:51.914029    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7953cc-9bbf-427e-a624-e53d759ba346" path="/var/lib/kubelet/pods/6a7953cc-9bbf-427e-a624-e53d759ba346/volumes"
Nov 02 15:16:52.633491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:52.633448    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.204:8443/health\": dial tcp 10.132.0.204:8443: connect: connection refused"
Nov 02 15:16:53.036085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:53.036051    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"]
Nov 02 15:16:53.042693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:53.042665    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a470704-tn2hc"]
Nov 02 15:16:53.890013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:53.889976    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40abc75-f698-4962-bc7b-539764eddf92" path="/var/lib/kubelet/pods/c40abc75-f698-4962-bc7b-539764eddf92/volumes"
Nov 02 15:16:55.047101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:55.047070    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"]
Nov 02 15:16:55.056173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:55.056147    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fedebb34-rhpww"]
Nov 02 15:16:55.979934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:55.979895    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb6e912-0ac6-46cf-9af4-53b713fc74b8" path="/var/lib/kubelet/pods/9eb6e912-0ac6-46cf-9af4-53b713fc74b8/volumes"
Nov 02 15:16:56.044692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:56.044545    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"]
Nov 02 15:16:56.052483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:56.051504    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2784718-n7zlg"]
Nov 02 15:16:56.827457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:56.827421    5041 generic.go:356] "Generic (PLEG): container finished" podID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53" exitCode=1
Nov 02 15:16:56.827665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:56.827494    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerDied","Data":"cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"}
Nov 02 15:16:56.827665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:56.827621    5041 scope.go:122] "RemoveContainer" containerID="7f81d88965fcc419911d4a00eaef73bba1f78dba0b6a00b6f212cd515df2e584"
Nov 02 15:16:56.828158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:56.828139    5041 scope.go:122] "RemoveContainer" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"
Nov 02 15:16:56.828638 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:16:56.828604    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm_odh-ai-gateway-infra(b26f0bf2-f8f7-416d-ba83-2c4b58c742b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"
Nov 02 15:16:57.890959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:16:57.890927    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb14608-6d60-4cb1-a855-b02c707b4f54" path="/var/lib/kubelet/pods/0eb14608-6d60-4cb1-a855-b02c707b4f54/volumes"
Nov 02 15:17:03.591385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:03.589442    5041 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:17:03.591385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:03.590001    5041 scope.go:122] "RemoveContainer" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"
Nov 02 15:17:03.591385 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:17:03.590443    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm_odh-ai-gateway-infra(b26f0bf2-f8f7-416d-ba83-2c4b58c742b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"
Nov 02 15:17:04.066918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:04.066885    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"]
Nov 02 15:17:04.072118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:04.072090    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddd553ee-hx82r"]
Nov 02 15:17:05.938383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:05.933909    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed8d345-257e-4bf2-9ac0-a14c29b4e00a" path="/var/lib/kubelet/pods/6ed8d345-257e-4bf2-9ac0-a14c29b4e00a/volumes"
Nov 02 15:17:06.770793 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:17:06.770747    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:17:10.053937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.053900    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"]
Nov 02 15:17:10.062395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.062330    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d675bd-6955r"]
Nov 02 15:17:10.464143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.464108    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"]
Nov 02 15:17:10.470280 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.470261    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:10.492296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.492274    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0a7163-dockercfg-qzljc\""
Nov 02 15:17:10.497088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.497064    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"]
Nov 02 15:17:10.502017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.501990    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0a7163\""
Nov 02 15:17:10.510803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.510772    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:10.510946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.510847    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrd25\" (UniqueName: \"kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25\") pod \"payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:10.843434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.811702    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:10.843434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.811819    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrd25\" (UniqueName: \"kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25\") pod \"payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:10.843434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.813707    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:10.882548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:10.882138    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrd25\" (UniqueName: \"kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25\") pod \"payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:11.085462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:11.085431    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:11.400904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:11.400866    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"]
Nov 02 15:17:11.402537 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:17:11.402511    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1332eff4_31e9_43b3_9454_413da38177f0.slice/crio-ffca5ef0342881e5739c66a73ef0373228a5574b373737c51254dc7b5989b63b WatchSource:0}: Error finding container ffca5ef0342881e5739c66a73ef0373228a5574b373737c51254dc7b5989b63b: Status 404 returned error can't find the container with id ffca5ef0342881e5739c66a73ef0373228a5574b373737c51254dc7b5989b63b
Nov 02 15:17:11.891047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:11.891016    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c68130-b369-406f-a46e-0ab6c08d56c8" path="/var/lib/kubelet/pods/69c68130-b369-406f-a46e-0ab6c08d56c8/volumes"
Nov 02 15:17:12.079022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:12.078987    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" event={"ID":"1332eff4-31e9-43b3-9454-413da38177f0","Type":"ContainerStarted","Data":"7d4610bd9d879feb7ed134689ace04c27f24668fd57782de878fa29834a16125"}
Nov 02 15:17:12.079210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:12.079040    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" event={"ID":"1332eff4-31e9-43b3-9454-413da38177f0","Type":"ContainerStarted","Data":"ffca5ef0342881e5739c66a73ef0373228a5574b373737c51254dc7b5989b63b"}
Nov 02 15:17:12.079270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:12.079247    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:12.131327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:12.131280    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" podStartSLOduration=2.131267328 podStartE2EDuration="2.131267328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:12.125461318 +0000 UTC m=+1284.383530769" watchObservedRunningTime="2026-11-02 15:17:12.131267328 +0000 UTC m=+1284.389336797"
Nov 02 15:17:15.544637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.544603    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"]
Nov 02 15:17:15.550907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.550887    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.573546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.573524    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ad276f-serving-cert\""
Nov 02 15:17:15.582513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.582490    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"]
Nov 02 15:17:15.624347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.624307    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tsnr\" (UniqueName: \"kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.624497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.624445    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.624568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.624550    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.624693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.624676    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.726676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.726269    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tsnr\" (UniqueName: \"kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.726676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.726383    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.726676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.726454    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.726676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.726499    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.727260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.727232    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.730108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.730082    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.730235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.730212    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.749228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.749208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tsnr\" (UniqueName: \"kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr\") pod \"maas-api-e2e-api-a-ad276f-597478b7f-29q7j\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.805999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.805936    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"]
Nov 02 15:17:15.811509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.811491    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.870198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.870164    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:15.940670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.919560    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-246137-openshift-default-dockercfg-hmh98\""
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.969780    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.969865    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sw8m\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.969958    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.969989    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.970057    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.970200    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.970338    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.970427    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:15.977390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:15.970514    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.064745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.064644    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"]
Nov 02 15:17:16.072819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.072782    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sw8m\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.072891    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.072924    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.072984    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.073136    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.073269    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.073339    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.073431    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.073514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.073477    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.074916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.074870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.075155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.075133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.082275    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"]
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.082885    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.083457    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.086211    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.086432    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.086574    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.092502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.092421    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"]
Nov 02 15:17:16.100699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.100673    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aba89bf-8sgp2"]
Nov 02 15:17:16.101983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.101964    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sw8m\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.110444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.108736    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0413335b-jcr5c"]
Nov 02 15:17:16.113111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.112273    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.390246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.390142    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:16.412916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.412886    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"]
Nov 02 15:17:16.415629 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:17:16.415599    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f09b710_9002_47ca_9ebf_20e32057732f.slice/crio-aa479d2439a68a090ce77c14e7a9bfe30beda418e8a15ca434192436dfa89076 WatchSource:0}: Error finding container aa479d2439a68a090ce77c14e7a9bfe30beda418e8a15ca434192436dfa89076: Status 404 returned error can't find the container with id aa479d2439a68a090ce77c14e7a9bfe30beda418e8a15ca434192436dfa89076
Nov 02 15:17:16.761463 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:17:16.761437    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05f53bc_3e54_4137_a7ab_ba0b6c06858a.slice/crio-bdfe0deed4ba209786417d1ec49af5c015d50ccae0e1138edbe2c5ecfe1cd057 WatchSource:0}: Error finding container bdfe0deed4ba209786417d1ec49af5c015d50ccae0e1138edbe2c5ecfe1cd057: Status 404 returned error can't find the container with id bdfe0deed4ba209786417d1ec49af5c015d50ccae0e1138edbe2c5ecfe1cd057
Nov 02 15:17:16.763121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.763087    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"]
Nov 02 15:17:16.764244 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.764209    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:16.764359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.764286    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:16.764359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:16.764334    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:17.122471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.120717    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"]
Nov 02 15:17:17.128663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.128640    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"]
Nov 02 15:17:17.128755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.128746    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.151348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.151326    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d7df16e-openshift-default-dockercfg-sn9kd\""
Nov 02 15:17:17.191381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191323    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.191567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191395    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.191567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191465    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjvg\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.191567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191505    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.191806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191567    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.191806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191589    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.191806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191797    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.192015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191848    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.192015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.191917    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.222251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.222212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" event={"ID":"e05f53bc-3e54-4137-a7ab-ba0b6c06858a","Type":"ContainerStarted","Data":"477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a"}
Nov 02 15:17:17.222565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.222538    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" event={"ID":"e05f53bc-3e54-4137-a7ab-ba0b6c06858a","Type":"ContainerStarted","Data":"bdfe0deed4ba209786417d1ec49af5c015d50ccae0e1138edbe2c5ecfe1cd057"}
Nov 02 15:17:17.227374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.227253    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j" event={"ID":"5f09b710-9002-47ca-9ebf-20e32057732f","Type":"ContainerStarted","Data":"dca4f56b0dea7dccdc5d89bd19be0094e98ef0c911af5d645cc4f92aa90392c8"}
Nov 02 15:17:17.227374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.227318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j" event={"ID":"5f09b710-9002-47ca-9ebf-20e32057732f","Type":"ContainerStarted","Data":"aa479d2439a68a090ce77c14e7a9bfe30beda418e8a15ca434192436dfa89076"}
Nov 02 15:17:17.284991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.284888    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podStartSLOduration=2.284872999 podStartE2EDuration="2.284872999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:17.278616558 +0000 UTC m=+1289.536686038" watchObservedRunningTime="2026-11-02 15:17:17.284872999 +0000 UTC m=+1289.542942448"
Nov 02 15:17:17.292962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.292928    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293002    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293219    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293284    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjvg\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293325    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293415    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293440    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293698 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293672    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.293859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.293835    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.294416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.294351    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.294578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.294496    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.294759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.294740    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.295472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.295449    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.296887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.296857    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.299803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.299782    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.363409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.352842    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjvg\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.363409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.354529    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j" podStartSLOduration=1.979836969 podStartE2EDuration="2.354511745s" totalImagesPullingTime="374.674776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:17.339488094 +0000 UTC m=+1289.597557820" watchObservedRunningTime="2026-11-02 15:17:17.354511745 +0000 UTC m=+1289.612581196"
Nov 02 15:17:17.392085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.391273    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:17.392085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.391864    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.506602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.502724    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:17.904886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.904801    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d2e9f6-a59f-4c41-ad1f-c282c1c669de" path="/var/lib/kubelet/pods/84d2e9f6-a59f-4c41-ad1f-c282c1c669de/volumes"
Nov 02 15:17:17.905583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:17.905557    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ea7178-3cbb-4f92-bb30-597ef2c0332c" path="/var/lib/kubelet/pods/91ea7178-3cbb-4f92-bb30-597ef2c0332c/volumes"
Nov 02 15:17:18.085266 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:17:18.085199    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa384528_cbbf_43e7_85ec_ca9c409d2b4e.slice/crio-a724e0f4fcfb509f375b609192081d3f6955fc36be9e176e304e034c95d94abb WatchSource:0}: Error finding container a724e0f4fcfb509f375b609192081d3f6955fc36be9e176e304e034c95d94abb: Status 404 returned error can't find the container with id a724e0f4fcfb509f375b609192081d3f6955fc36be9e176e304e034c95d94abb
Nov 02 15:17:18.125864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.125815    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:18.126167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.126139    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:18.126321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.126304    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:18.173665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.163494    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"]
Nov 02 15:17:18.257190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.256790    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" event={"ID":"fa384528-cbbf-43e7-85ec-ca9c409d2b4e","Type":"ContainerStarted","Data":"a724e0f4fcfb509f375b609192081d3f6955fc36be9e176e304e034c95d94abb"}
Nov 02 15:17:18.257190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.257094    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:18.397070 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.397036    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:17:18.397295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.397103    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:17:18.881172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:18.881129    5041 scope.go:122] "RemoveContainer" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"
Nov 02 15:17:19.287661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:19.287626    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" event={"ID":"fa384528-cbbf-43e7-85ec-ca9c409d2b4e","Type":"ContainerStarted","Data":"145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa"}
Nov 02 15:17:19.392688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:19.391184    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.392688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:19.391240    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:17:19.507239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:19.507152    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:19.508152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:19.508122    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.508272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:19.508174    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:17:20.297882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.297851    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerStarted","Data":"8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20"}
Nov 02 15:17:20.298495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.298466    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:17:20.367438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.365253    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podStartSLOduration=4.365230698 podStartE2EDuration="4.365230698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:19.338722739 +0000 UTC m=+1291.596792191" watchObservedRunningTime="2026-11-02 15:17:20.365230698 +0000 UTC m=+1292.623300148"
Nov 02 15:17:20.391595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.391566    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.391776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.391614    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:17:20.507289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.507258    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.507471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:20.507308    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:17:21.067963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.067907    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"]
Nov 02 15:17:21.073622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.073589    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fddf9a-9wtf2"]
Nov 02 15:17:21.395425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.391455    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.395425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.391520    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:17:21.507008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.506973    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.507175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.507026    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:17:21.983228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:21.983193    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fcfacc-200c-4487-b4dd-6e6d6eedfcb4" path="/var/lib/kubelet/pods/48fcfacc-200c-4487-b4dd-6e6d6eedfcb4/volumes"
Nov 02 15:17:22.390971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:22.390890    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.390971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:22.390955    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:17:22.507283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:22.507241    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.507752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:22.507454    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:17:23.091811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:23.091773    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:17:23.391132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:23.391053    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.391346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:23.391323    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:17:23.507438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:23.507401    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.507806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:23.507454    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:17:24.294473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:24.294440    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:17:24.397439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:24.397405    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:24.398719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:24.398694    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:24.399456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:24.399430    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:17:24.508748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:24.508682    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.509332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:24.509293    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:17:25.518677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:25.518645    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:25.519222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:25.519197    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:25.520256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:25.520228    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:17:26.046420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:26.046358    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"]
Nov 02 15:17:26.055416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:26.055385    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324cc797-zw7mq"]
Nov 02 15:17:26.347588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:26.347509    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:17:27.893185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:27.893154    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64b5dc2-3b7a-4feb-861e-f3c78bba9583" path="/var/lib/kubelet/pods/b64b5dc2-3b7a-4feb-861e-f3c78bba9583/volumes"
Nov 02 15:17:36.045746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:36.045227    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"]
Nov 02 15:17:36.055132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:36.055097    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9d4011-s6s8c"]
Nov 02 15:17:36.905448 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:17:36.905415    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:17:38.010674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:38.010636    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1384b9b-ac30-4834-b37d-b013db4b294f" path="/var/lib/kubelet/pods/b1384b9b-ac30-4834-b37d-b013db4b294f/volumes"
Nov 02 15:17:40.697387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:40.696698    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"]
Nov 02 15:17:40.697940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:40.697553    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy" containerID="cri-o://c18b802140dbdcf147a0d57c3824b1f6d61cae3b2030a5345327ee89b2bf7309" gracePeriod=30
Nov 02 15:17:45.907507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:45.907467    5041 generic.go:356] "Generic (PLEG): container finished" podID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerID="c18b802140dbdcf147a0d57c3824b1f6d61cae3b2030a5345327ee89b2bf7309" exitCode=0
Nov 02 15:17:45.918127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:45.918084    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" event={"ID":"0e3d7955-073c-4271-83f7-d18f2962b0de","Type":"ContainerDied","Data":"c18b802140dbdcf147a0d57c3824b1f6d61cae3b2030a5345327ee89b2bf7309"}
Nov 02 15:17:46.110093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.110069    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:17:46.276805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.276716    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.276999    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277168    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277228    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277295    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277296    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.277346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277330    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277451    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277536    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277573    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket\") pod \"0e3d7955-073c-4271-83f7-d18f2962b0de\" (UID: \"0e3d7955-073c-4271-83f7-d18f2962b0de\") "
Nov 02 15:17:46.277670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.277648    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.278389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.278334    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.278716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.278680    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.279423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.279394    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.279591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.279520    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.280114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280075    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.280220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280195    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3d7955-073c-4271-83f7-d18f2962b0de-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.280220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280211    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.280589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280153    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.280589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280224    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.280767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280593    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.280767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280610    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.280767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.280738    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.281662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.281635    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x" pod "0e3d7955-073c-4271-83f7-d18f2962b0de" (UID: "0e3d7955-073c-4271-83f7-d18f2962b0de"). InnerVolumeSpecName "kube-api-access-z6q4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.382313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.382278    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6q4x\" (UniqueName: \"kubernetes.io/projected/0e3d7955-073c-4271-83f7-d18f2962b0de-kube-api-access-z6q4x\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.382313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.382310    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.382313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.382319    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3d7955-073c-4271-83f7-d18f2962b0de-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.931663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.931634    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"
Nov 02 15:17:46.932211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.931658    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h" event={"ID":"0e3d7955-073c-4271-83f7-d18f2962b0de","Type":"ContainerDied","Data":"ea421e9866eb0c2b297e95a4b7452832c4575e7eef98b785f2357fe076c691d9"}
Nov 02 15:17:46.932211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.931742    5041 scope.go:122] "RemoveContainer" containerID="c18b802140dbdcf147a0d57c3824b1f6d61cae3b2030a5345327ee89b2bf7309"
Nov 02 15:17:46.990138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.990110    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"]
Nov 02 15:17:46.999228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:46.999201    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-2bk2h"]
Nov 02 15:17:47.939989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:47.939949    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" path="/var/lib/kubelet/pods/0e3d7955-073c-4271-83f7-d18f2962b0de/volumes"
Nov 02 15:17:48.035351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:48.035321    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"]
Nov 02 15:17:48.045150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:48.045125    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1030fd3-ngqf4"]
Nov 02 15:17:49.789252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.789219    5041 scope.go:122] "RemoveContainer" containerID="2b9a6b3a644bead7985328fddb98134c6ebb0d9fb8c067a651f583304fc52a7a"
Nov 02 15:17:49.801026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.801003    5041 scope.go:122] "RemoveContainer" containerID="fc8af10c95f7d67278cc5c0ffb93991e78529fdf3120275851a2d3fcebdd2c24"
Nov 02 15:17:49.811951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.811933    5041 scope.go:122] "RemoveContainer" containerID="f4ac1f1cc1abea325ce14aa666f1b08714b33fd6614b2c37a33f4eaf7b24152f"
Nov 02 15:17:49.823009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.822990    5041 scope.go:122] "RemoveContainer" containerID="09b4005651dedb075dfe0f1cc640704f449e9fed97bf82955360a7e2590b04e8"
Nov 02 15:17:49.834426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.834407    5041 scope.go:122] "RemoveContainer" containerID="2ed9d2e3d043a1f281e35751e47f5331b7c1f1a78648ec1d7d090155d650cf4c"
Nov 02 15:17:49.844938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.844920    5041 scope.go:122] "RemoveContainer" containerID="73a5fc0da003b8f9d534a108863cb93b87360b9bb1ab2714b7d64ceb5aee4e66"
Nov 02 15:17:49.855101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.855081    5041 scope.go:122] "RemoveContainer" containerID="0d130197fd6ad75d94c9c616fb1ccc4d25b106f2a25aa4692e8c8bae2603ea3a"
Nov 02 15:17:49.873047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.873026    5041 scope.go:122] "RemoveContainer" containerID="fe620a5e9dbb0f9f71d32e235c45438c5a0907e212ac15fb78b9103115c5b79b"
Nov 02 15:17:49.884626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.884604    5041 scope.go:122] "RemoveContainer" containerID="8fab1f42347e4c20d5dd72325ad86d743e235842cc5644b479280f046df051d1"
Nov 02 15:17:49.893700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.893672    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc21ddd-b54b-4848-ba15-1779ea2ebee0" path="/var/lib/kubelet/pods/8bc21ddd-b54b-4848-ba15-1779ea2ebee0/volumes"
Nov 02 15:17:49.895884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.895869    5041 scope.go:122] "RemoveContainer" containerID="cac62fa4d43def8b03d8e726661de7611a4c29d03877a8b4195752f7918b44de"
Nov 02 15:17:49.906626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.906605    5041 scope.go:122] "RemoveContainer" containerID="93ae1c5fda5d6c63332fa7c3cc957bcccdceb718fd5a860e870852974f416487"
Nov 02 15:17:49.916500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.916481    5041 scope.go:122] "RemoveContainer" containerID="ee0277b70ad257c8f088448586eb1940901bdf24831e090dd10c89a4f5371043"
Nov 02 15:17:49.926358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.926337    5041 scope.go:122] "RemoveContainer" containerID="27c302a2945feff4086c3e25af8c3746958635971a0e62236f6a22d3d96687f9"
Nov 02 15:17:49.936095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:17:49.936079    5041 scope.go:122] "RemoveContainer" containerID="f0a530bd35e39e7e5c345aae3dac896efb771eed8d87fff2c141af86a758c9ba"
Nov 02 15:18:00.962708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:00.962674    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"]
Nov 02 15:18:00.963630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:00.963613    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy"
Nov 02 15:18:00.963701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:00.963631    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy"
Nov 02 15:18:00.963812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:00.963802    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3d7955-073c-4271-83f7-d18f2962b0de" containerName="istio-proxy"
Nov 02 15:18:00.969857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:00.969837    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:00.993028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:00.993000    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"]
Nov 02 15:18:01.115445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115409    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.115624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115479    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.115720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115700    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.115791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115748    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.115859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115842    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nhc\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.115911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115878    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.115968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.115951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.116068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.116052    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.116123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.116103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218320    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218425    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218484    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218539    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218852    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nhc\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218938    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.218989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.218978    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.219606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.219576    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.220509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.220474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.220628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.220579    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.220954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.220931    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.221075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.221056    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.223131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.223105    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.223434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.223343    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.255960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.255938    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nhc\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.268510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.268489    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.287511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.287485    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:01.628200 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:18:01.628170    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f6ea88_81e7_4b5f_82e9_c08acfb45ff2.slice/crio-c8ad5c8c756669cf54cc61b60179af48457af3d804bbb559e431eb1307ffb0d4 WatchSource:0}: Error finding container c8ad5c8c756669cf54cc61b60179af48457af3d804bbb559e431eb1307ffb0d4: Status 404 returned error can't find the container with id c8ad5c8c756669cf54cc61b60179af48457af3d804bbb559e431eb1307ffb0d4
Nov 02 15:18:01.629520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.629493    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"]
Nov 02 15:18:01.630706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.630677    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:01.630759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.630740    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:01.630798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:01.630767    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.026988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.026949    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"]
Nov 02 15:18:02.032798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.032775    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.039048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.039010    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.039179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.039053    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.039241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.039225    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4xt\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.039410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.039379    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.041543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.041515    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.041664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.041596    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.041734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.041712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.041794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.041745    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.041794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.041781    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.125589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.101497    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"]
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.147823    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.148750    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.149431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.148914    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.149529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.149941    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.150090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.150601    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4xt\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.150923    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.152089    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.152436    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.151410    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.152912    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.153259    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.156792    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.173386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.157416    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.201929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.201884    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.222649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.222304    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4xt\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.272428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.267940    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" event={"ID":"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2","Type":"ContainerStarted","Data":"b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035"}
Nov 02 15:18:02.272428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.267995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" event={"ID":"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2","Type":"ContainerStarted","Data":"c8ad5c8c756669cf54cc61b60179af48457af3d804bbb559e431eb1307ffb0d4"}
Nov 02 15:18:02.288637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.288456    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:02.288996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.288965    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:18:02.289133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.289018    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:18:02.327861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.325220    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podStartSLOduration=2.325200891 podStartE2EDuration="2.325200891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:02.317754899 +0000 UTC m=+1334.575824375" watchObservedRunningTime="2026-11-02 15:18:02.325200891 +0000 UTC m=+1334.583270341"
Nov 02 15:18:02.353101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.352782    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:02.778391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.765566    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"]
Nov 02 15:18:02.809492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.807779    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.809492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.807887    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.809492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:02.807939    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:03.283191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.283155    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" event={"ID":"5d260c65-7fb3-4dbc-96b8-e1a60c44e655","Type":"ContainerStarted","Data":"e5545b813ad2f22bc08a4ed3a030a4a3058ed937badc3c9e82cde24c20ba6278"}
Nov 02 15:18:03.283622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.283215    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" event={"ID":"5d260c65-7fb3-4dbc-96b8-e1a60c44e655","Type":"ContainerStarted","Data":"93a1d2b57e728771870597fc60a6ebb67081ec1cb456ce9eec0c2d081ce1db4f"}
Nov 02 15:18:03.288760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.288733    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:18:03.288891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.288782    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:18:03.348407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.348339    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podStartSLOduration=2.348322459 podStartE2EDuration="2.348322459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:03.328727034 +0000 UTC m=+1335.586796485" watchObservedRunningTime="2026-11-02 15:18:03.348322459 +0000 UTC m=+1335.606391909"
Nov 02 15:18:03.354098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.354065    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:03.355423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.355398    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:03.355526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:03.355448    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:04.288083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:04.288044    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:18:04.288541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:04.288093    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:18:04.365797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:04.365357    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:04.365797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:04.365487    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:05.288607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:05.288569    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:18:05.288978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:05.288624    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:18:05.353747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:05.353708    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:05.354035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:05.354004    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:06.297090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:06.296951    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:18:06.297786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:06.297746    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:18:06.353815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:06.353772    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:06.354124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:06.354090    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:06.795809 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:18:06.795773    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:18:07.288623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:07.288588    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:18:07.288798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:07.288645    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:18:07.353772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:07.353723    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:07.354381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:07.354325    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:08.300190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:08.300159    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:08.301927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:08.301886    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:08.308856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:08.308834    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:18:08.353417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:08.353390    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:08.353596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:08.353572    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:09.353808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:09.353763    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:18:09.354196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:09.353835    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:18:10.358579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:10.358553    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:10.358946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:10.358898    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:10.359600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:10.359583    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:18:17.150226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.150181    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"]
Nov 02 15:18:17.157533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.157507    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.170670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.170635    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"]
Nov 02 15:18:17.210030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.209996    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.210232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210152    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.210232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210220    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.211918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.211918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210319    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.211918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210345    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.211918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210471    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cbn\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.211918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210603    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.211918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.210632    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.311740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.311682    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.311740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.311732    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.311835    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.311981    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312050    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312093    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312150    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312180    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312295    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cbn\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312308    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312615    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312649    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.312794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.312778    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.313056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.313023    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.315577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.315555    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.315577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.315569    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.334400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.333428    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.337838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.337774    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cbn\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:17.479098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:17.479067    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:18.224661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.224592    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:18.225403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.225358    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:18.225626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.225592    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:18.253671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.253615    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"]
Nov 02 15:18:18.796549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.789175    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" event={"ID":"d062b215-806e-4bef-a3cf-a5f42a1492e4","Type":"ContainerStarted","Data":"bd3b7c106d4c36c53a87362d39fdb4fd0b102ee8ef784bad6dce83a5e91e42e4"}
Nov 02 15:18:18.796549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.789247    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" event={"ID":"d062b215-806e-4bef-a3cf-a5f42a1492e4","Type":"ContainerStarted","Data":"2d88cf558a3f486f330940578b14f5ede73a0710d2f348df9dd49ff28344c36a"}
Nov 02 15:18:18.921385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:18.906650    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podStartSLOduration=1.906630513 podStartE2EDuration="1.906630513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:18.883953653 +0000 UTC m=+1351.142023124" watchObservedRunningTime="2026-11-02 15:18:18.906630513 +0000 UTC m=+1351.164699963"
Nov 02 15:18:19.508788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:19.508756    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:19.633022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:19.632695    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:19.633022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:19.632765    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:20.479747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:20.479705    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:20.479938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:20.479769    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:21.491388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:21.482849    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:21.491388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:21.482934    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:22.479623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:22.479583    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:22.479815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:22.479641    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:23.480228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:23.480184    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:23.480667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:23.480239    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:24.483814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:24.483776    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:24.484425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:24.484392    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:25.480454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:25.480413    5041 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 15:18:25.480619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:25.480468    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 15:18:26.623548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:26.623511    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:26.624031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:26.623879    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:26.624846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:26.624823    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:18:36.786189 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:18:36.786153    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:18:48.750792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:48.750750    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"]
Nov 02 15:18:48.787233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:48.787202    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"]
Nov 02 15:18:50.064973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.064939    5041 scope.go:122] "RemoveContainer" containerID="22f3e863285a24f912fb9e1c3fbebc81382b7b8dd2738b232627869da9f43acc"
Nov 02 15:18:50.078125 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.076976    5041 scope.go:122] "RemoveContainer" containerID="02d8db49a6e88380dd67cc651d9502f5394a874ee9be11d90b5fbfc927abfacf"
Nov 02 15:18:50.093978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.093959    5041 scope.go:122] "RemoveContainer" containerID="6fb3b06156b69ec63100d6cbc179db28453f01b099dc9a81e7e18fa62d84c93e"
Nov 02 15:18:50.104236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.104215    5041 scope.go:122] "RemoveContainer" containerID="82c92272bbd6bd2ba9f5a10d0eb389ecc36425b2ae7aba54aaaeb94dc6e9e0e3"
Nov 02 15:18:50.114489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.114467    5041 scope.go:122] "RemoveContainer" containerID="b8bdef89d623d2a5186660ffd3b34230ef629af7ac816024005eaa50d71a6912"
Nov 02 15:18:50.124666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.124645    5041 scope.go:122] "RemoveContainer" containerID="ee4ae6f760302dc0fc9857b8ed8b7ab2cb1e01069d3cbd391de83f136ba90670"
Nov 02 15:18:50.134638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.134617    5041 scope.go:122] "RemoveContainer" containerID="d1c3d351e031ac0b30a6a2bcdb79e3aaa7749aa90653c555c8644b113cb2a132"
Nov 02 15:18:50.147031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.147010    5041 scope.go:122] "RemoveContainer" containerID="bcec9ef43f6a2076e809d09a8a90743e35bde71e47968534ee1e9d2548e54b6c"
Nov 02 15:18:50.157901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.157881    5041 scope.go:122] "RemoveContainer" containerID="11756cfa99147cf9d911048ee3436fb0693fbd3ec4961596ddc8fabce40fd44f"
Nov 02 15:18:50.168139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.168113    5041 scope.go:122] "RemoveContainer" containerID="bd25c6fadc197d77930b93d97e2cfd2525c8fef7030736db2693850a4c6e2d62"
Nov 02 15:18:50.178014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:50.177984    5041 scope.go:122] "RemoveContainer" containerID="dea82b4aef903ac71cc7fd74308400ecc44b431198b8e8554f4ed84cab3a3846"
Nov 02 15:18:53.751709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:53.751657    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb" podUID="ab3f400f-be12-41bc-86a6-f0a1723ea179" containerName="payload-processing" containerID="cri-o://2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64" gracePeriod=25
Nov 02 15:18:53.787675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:53.787618    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5" podUID="e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" containerName="payload-pre-processing" containerID="cri-o://c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2" gracePeriod=25
Nov 02 15:18:54.152413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.152339    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:18:54.159416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.158217    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:18:54.263460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.259986    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume\") pod \"ab3f400f-be12-41bc-86a6-f0a1723ea179\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") "
Nov 02 15:18:54.263460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.260271    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz\" (UniqueName: \"kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz\") pod \"ab3f400f-be12-41bc-86a6-f0a1723ea179\" (UID: \"ab3f400f-be12-41bc-86a6-f0a1723ea179\") "
Nov 02 15:18:54.263460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.260434    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume\") pod \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") "
Nov 02 15:18:54.263460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.260514    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v\" (UniqueName: \"kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v\") pod \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\" (UID: \"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719\") "
Nov 02 15:18:54.284821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.283562    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz" pod "ab3f400f-be12-41bc-86a6-f0a1723ea179" (UID: "ab3f400f-be12-41bc-86a6-f0a1723ea179"). InnerVolumeSpecName "kube-api-access-rlphz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:54.284821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.284059    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume" pod "ab3f400f-be12-41bc-86a6-f0a1723ea179" (UID: "ab3f400f-be12-41bc-86a6-f0a1723ea179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:54.284821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.284564    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume" pod "e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" (UID: "e9c8f6d6-8f77-4b25-a3ed-2b50602b3719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:54.291445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.290635    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v" pod "e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" (UID: "e9c8f6d6-8f77-4b25-a3ed-2b50602b3719"). InnerVolumeSpecName "kube-api-access-9h45v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:54.363649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.363570    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlphz\" (UniqueName: \"kubernetes.io/projected/ab3f400f-be12-41bc-86a6-f0a1723ea179-kube-api-access-rlphz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:18:54.363649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.363599    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:18:54.363649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.363609    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h45v\" (UniqueName: \"kubernetes.io/projected/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719-kube-api-access-9h45v\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:18:54.363649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.363618    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f400f-be12-41bc-86a6-f0a1723ea179-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:18:54.790140 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.790105    5041 generic.go:356] "Generic (PLEG): container finished" podID="e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" containerID="c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2" exitCode=0
Nov 02 15:18:54.790614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.790144    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"
Nov 02 15:18:54.790614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.790190    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5" event={"ID":"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719","Type":"ContainerDied","Data":"c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2"}
Nov 02 15:18:54.790614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.790239    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5" event={"ID":"e9c8f6d6-8f77-4b25-a3ed-2b50602b3719","Type":"ContainerDied","Data":"89a9485647e777ef85b3ebd4b57b52ca554ba37bd0267308fe48df4294480e9b"}
Nov 02 15:18:54.790614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.790255    5041 scope.go:122] "RemoveContainer" containerID="c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2"
Nov 02 15:18:54.791909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.791789    5041 generic.go:356] "Generic (PLEG): container finished" podID="ab3f400f-be12-41bc-86a6-f0a1723ea179" containerID="2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64" exitCode=0
Nov 02 15:18:54.791909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.791836    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"
Nov 02 15:18:54.791909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.791877    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb" event={"ID":"ab3f400f-be12-41bc-86a6-f0a1723ea179","Type":"ContainerDied","Data":"2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64"}
Nov 02 15:18:54.792078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.791946    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb" event={"ID":"ab3f400f-be12-41bc-86a6-f0a1723ea179","Type":"ContainerDied","Data":"a55a6ec5ceb9225f9377fce98533591e55a88a3ce056115014b2ffe98b794896"}
Nov 02 15:18:54.803999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.803979    5041 scope.go:122] "RemoveContainer" containerID="c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2"
Nov 02 15:18:54.804269 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:18:54.804248    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2\": container with ID starting with c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2 not found: ID does not exist" containerID="c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2"
Nov 02 15:18:54.804308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.804277    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2"} err="failed to get container status \"c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2\": rpc error: code = NotFound desc = could not find container \"c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2\": container with ID starting with c75fce68435b1cf4417f67d5a5abd4e49ed91f97c93c01b92d8332a1b6a56cc2 not found: ID does not exist"
Nov 02 15:18:54.804308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.804293    5041 scope.go:122] "RemoveContainer" containerID="2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64"
Nov 02 15:18:54.815201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.815183    5041 scope.go:122] "RemoveContainer" containerID="2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64"
Nov 02 15:18:54.815469 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:18:54.815450    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64\": container with ID starting with 2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64 not found: ID does not exist" containerID="2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64"
Nov 02 15:18:54.815534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.815475    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64"} err="failed to get container status \"2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64\": rpc error: code = NotFound desc = could not find container \"2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64\": container with ID starting with 2d63c3d7341b8585ff1b1b93d2e0179bfd65a553095e57295209b07c5ae11b64 not found: ID does not exist"
Nov 02 15:18:54.849805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.849782    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"]
Nov 02 15:18:54.859255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.859229    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7afd1a6-6574598bbc-w7xb5"]
Nov 02 15:18:54.885674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.885652    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"]
Nov 02 15:18:54.891878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:54.891856    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7afd1a6-564dd68fd9-qz9rb"]
Nov 02 15:18:55.527314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:55.527274    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"]
Nov 02 15:18:55.527571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:55.527531    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy" containerID="cri-o://8d4c3aa74446848dfeead771c8ef4dc2a7e64375383f27470a9f3cf9c653f828" gracePeriod=30
Nov 02 15:18:56.160694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.160650    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3f400f-be12-41bc-86a6-f0a1723ea179" path="/var/lib/kubelet/pods/ab3f400f-be12-41bc-86a6-f0a1723ea179/volumes"
Nov 02 15:18:56.162144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.161623    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" path="/var/lib/kubelet/pods/e9c8f6d6-8f77-4b25-a3ed-2b50602b3719/volumes"
Nov 02 15:18:56.162621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.162230    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"]
Nov 02 15:18:56.162621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.162341    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"]
Nov 02 15:18:56.162824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.162722    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy" containerID="cri-o://756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36" gracePeriod=30
Nov 02 15:18:56.168672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.168605    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy" containerID="cri-o://492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7" gracePeriod=30
Nov 02 15:18:56.170885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.170859    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"]
Nov 02 15:18:56.171098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:18:56.171061    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy" containerID="cri-o://bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad" gracePeriod=30
Nov 02 15:19:00.657473 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:00.657430    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3604113c_a16f_42b0_b65e_f0e3456c6156.slice/crio-conmon-8d4c3aa74446848dfeead771c8ef4dc2a7e64375383f27470a9f3cf9c653f828.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:19:00.954384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:00.953646    5041 generic.go:356] "Generic (PLEG): container finished" podID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerID="8d4c3aa74446848dfeead771c8ef4dc2a7e64375383f27470a9f3cf9c653f828" exitCode=0
Nov 02 15:19:00.954384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:00.953891    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" event={"ID":"3604113c-a16f-42b0-b65e-f0e3456c6156","Type":"ContainerDied","Data":"8d4c3aa74446848dfeead771c8ef4dc2a7e64375383f27470a9f3cf9c653f828"}
Nov 02 15:19:01.491133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.490995    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:19:01.627078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627034    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627097    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627146    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627249    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627287    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627392    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627446    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627779    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.627971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.627942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket\") pod \"3604113c-a16f-42b0-b65e-f0e3456c6156\" (UID: \"3604113c-a16f-42b0-b65e-f0e3456c6156\") "
Nov 02 15:19:01.629436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.629397    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.629694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.629661    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.629910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.629833    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.630082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.630040    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.630349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.630328    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.633458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.633430    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.633627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.633602    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "kube-api-access-hn7h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.635235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.635210    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.635723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.635683    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy" pod "3604113c-a16f-42b0-b65e-f0e3456c6156" (UID: "3604113c-a16f-42b0-b65e-f0e3456c6156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.709448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.694882    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:19:01.730452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730425    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7h6\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-kube-api-access-hn7h6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730457    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730473    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604113c-a16f-42b0-b65e-f0e3456c6156-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730484    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730496    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730508    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730522    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730534    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604113c-a16f-42b0-b65e-f0e3456c6156-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.730572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.730548    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604113c-a16f-42b0-b65e-f0e3456c6156-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.788435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.779884    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:19:01.788648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.788589    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:19:01.832973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.832937    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833026    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833058    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.833168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833152    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833182    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.833334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833225    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.833334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833267    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.833334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833303    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833346    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.833570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833426    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.833570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833453    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.833570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833500    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.833570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833553    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.833824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833626    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.833824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833676    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833805    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833847    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.833988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833880    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.833988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833965    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.834136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.833996    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.834136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834028    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.834136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834115    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.834294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834156    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.834294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834206    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.834419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834319    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token\") pod \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\" (UID: \"6a3c9823-4ec5-443a-86a2-dba33a1921cd\") "
Nov 02 15:19:01.834419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834397    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert\") pod \"03a6879a-f491-4692-82c5-6474fafae08d\" (UID: \"03a6879a-f491-4692-82c5-6474fafae08d\") "
Nov 02 15:19:01.834530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.834437    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data\") pod \"0e259152-910d-43c6-a8fb-5b231bb2632c\" (UID: \"0e259152-910d-43c6-a8fb-5b231bb2632c\") "
Nov 02 15:19:01.836434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.836401    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.836853    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.837746    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.838064    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.838357    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.838655    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.840596    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.840768    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.840910    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.840999    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.841154    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.842088    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.842216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.842140    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.844386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.843913    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.844386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.844045    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.844386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.844062    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.845721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.845680    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.845943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.845911    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.850220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.850194    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863517    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "kube-api-access-5vg66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863554    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863560    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "kube-api-access-rrnzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863615    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863643    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz" pod "03a6879a-f491-4692-82c5-6474fafae08d" (UID: "03a6879a-f491-4692-82c5-6474fafae08d"). InnerVolumeSpecName "kube-api-access-k96cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863670    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token" pod "0e259152-910d-43c6-a8fb-5b231bb2632c" (UID: "0e259152-910d-43c6-a8fb-5b231bb2632c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.863824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.863792    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.865584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.865535    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token" pod "6a3c9823-4ec5-443a-86a2-dba33a1921cd" (UID: "6a3c9823-4ec5-443a-86a2-dba33a1921cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.938420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938391    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938419    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k96cz\" (UniqueName: \"kubernetes.io/projected/03a6879a-f491-4692-82c5-6474fafae08d-kube-api-access-k96cz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938432    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938442    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938455    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938465    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938476    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938484    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938492    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938500    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938508    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vg66\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-kube-api-access-5vg66\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938516    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938529    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrnzm\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-kube-api-access-rrnzm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938540    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938548    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938557    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938566    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938573    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a6879a-f491-4692-82c5-6474fafae08d-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938581    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938591    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3c9823-4ec5-443a-86a2-dba33a1921cd-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938601    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938613    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e259152-910d-43c6-a8fb-5b231bb2632c-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938622    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938631    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a6879a-f491-4692-82c5-6474fafae08d-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938639    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3c9823-4ec5-443a-86a2-dba33a1921cd-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938647    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a6879a-f491-4692-82c5-6474fafae08d-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.938646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.938655    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e259152-910d-43c6-a8fb-5b231bb2632c-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.965159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.965112    5041 generic.go:356] "Generic (PLEG): container finished" podID="03a6879a-f491-4692-82c5-6474fafae08d" containerID="756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36" exitCode=0
Nov 02 15:19:01.965159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.965159    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"
Nov 02 15:19:01.965394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.965199    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" event={"ID":"03a6879a-f491-4692-82c5-6474fafae08d","Type":"ContainerDied","Data":"756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36"}
Nov 02 15:19:01.965394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.965248    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd" event={"ID":"03a6879a-f491-4692-82c5-6474fafae08d","Type":"ContainerDied","Data":"d8e8a1fd1b3b7ec9eb1409383e38cb6492f68019163216f4b3252006edc130ae"}
Nov 02 15:19:01.965394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.965275    5041 scope.go:122] "RemoveContainer" containerID="756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36"
Nov 02 15:19:01.967233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.967215    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"
Nov 02 15:19:01.967351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.967263    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2" event={"ID":"3604113c-a16f-42b0-b65e-f0e3456c6156","Type":"ContainerDied","Data":"79b982de66ba95acbe232ff92d5b835594453ce5e264d0802e7c120f96fcdf8e"}
Nov 02 15:19:01.969342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.969297    5041 generic.go:356] "Generic (PLEG): container finished" podID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerID="492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7" exitCode=0
Nov 02 15:19:01.969505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.969348    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"
Nov 02 15:19:01.969505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.969459    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" event={"ID":"0e259152-910d-43c6-a8fb-5b231bb2632c","Type":"ContainerDied","Data":"492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7"}
Nov 02 15:19:01.969632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.969534    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws" event={"ID":"0e259152-910d-43c6-a8fb-5b231bb2632c","Type":"ContainerDied","Data":"af3c2fa614c6750b79d52cfc07bd5d85534d7bd0ae994bcd2e7ddc21356aaf19"}
Nov 02 15:19:01.971313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.971294    5041 generic.go:356] "Generic (PLEG): container finished" podID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerID="bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad" exitCode=0
Nov 02 15:19:01.971406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.971335    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"
Nov 02 15:19:01.971406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.971347    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" event={"ID":"6a3c9823-4ec5-443a-86a2-dba33a1921cd","Type":"ContainerDied","Data":"bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad"}
Nov 02 15:19:01.971406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.971394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw" event={"ID":"6a3c9823-4ec5-443a-86a2-dba33a1921cd","Type":"ContainerDied","Data":"ec0884d01d14778179011f08ca62b7478097b8b36717f6f042ab693b81e26320"}
Nov 02 15:19:01.979275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.979252    5041 scope.go:122] "RemoveContainer" containerID="756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36"
Nov 02 15:19:01.979600 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:01.979579    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36\": container with ID starting with 756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36 not found: ID does not exist" containerID="756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36"
Nov 02 15:19:01.979686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.979607    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36"} err="failed to get container status \"756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36\": rpc error: code = NotFound desc = could not find container \"756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36\": container with ID starting with 756a1a6c001761f6983ebcee5e43ab81ee4624a7e1d25249545dd2503b6bac36 not found: ID does not exist"
Nov 02 15:19:01.979686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:01.979625    5041 scope.go:122] "RemoveContainer" containerID="8d4c3aa74446848dfeead771c8ef4dc2a7e64375383f27470a9f3cf9c653f828"
Nov 02 15:19:02.017458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.016530    5041 scope.go:122] "RemoveContainer" containerID="492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7"
Nov 02 15:19:02.080389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.072732    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"]
Nov 02 15:19:02.195566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.195530    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-ck7ws"]
Nov 02 15:19:02.204438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.204404    5041 scope.go:122] "RemoveContainer" containerID="492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7"
Nov 02 15:19:02.221983 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:02.221946    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7\": container with ID starting with 492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7 not found: ID does not exist" containerID="492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7"
Nov 02 15:19:02.222226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.222199    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7"} err="failed to get container status \"492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7\": rpc error: code = NotFound desc = could not find container \"492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7\": container with ID starting with 492d3ea750f07053ad9b947d08f001496298ff1cb510d65607edec4484cf6ef7 not found: ID does not exist"
Nov 02 15:19:02.222340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.222326    5041 scope.go:122] "RemoveContainer" containerID="bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad"
Nov 02 15:19:02.295546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.295456    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"]
Nov 02 15:19:02.342782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.342748    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-hwzm2"]
Nov 02 15:19:02.357212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.357167    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"]
Nov 02 15:19:02.357461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.357235    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pfkvd"]
Nov 02 15:19:02.367728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.367686    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"]
Nov 02 15:19:02.368908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.368885    5041 scope.go:122] "RemoveContainer" containerID="bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad"
Nov 02 15:19:02.369272 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:02.369251    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad\": container with ID starting with bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad not found: ID does not exist" containerID="bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad"
Nov 02 15:19:02.369405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.369278    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad"} err="failed to get container status \"bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad\": rpc error: code = NotFound desc = could not find container \"bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad\": container with ID starting with bd9a039e3763a95d0b15e7e0f4e6356e42b3a8bae2b5aa05a5169ed1cc7632ad not found: ID does not exist"
Nov 02 15:19:02.373670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:02.373645    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-dshmw"]
Nov 02 15:19:03.893748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:03.893716    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a6879a-f491-4692-82c5-6474fafae08d" path="/var/lib/kubelet/pods/03a6879a-f491-4692-82c5-6474fafae08d/volumes"
Nov 02 15:19:03.894282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:03.894265    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" path="/var/lib/kubelet/pods/0e259152-910d-43c6-a8fb-5b231bb2632c/volumes"
Nov 02 15:19:03.894902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:03.894885    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" path="/var/lib/kubelet/pods/3604113c-a16f-42b0-b65e-f0e3456c6156/volumes"
Nov 02 15:19:03.895701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:03.895685    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" path="/var/lib/kubelet/pods/6a3c9823-4ec5-443a-86a2-dba33a1921cd/volumes"
Nov 02 15:19:05.208334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:05.208294    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"]
Nov 02 15:19:05.208830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:05.208526    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy" containerID="cri-o://0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f" gracePeriod=30
Nov 02 15:19:06.771627 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:06.771585    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:19:10.522447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.522421    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:19:10.557659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.557630    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.557809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.557709    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.557809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.557763    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.557809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.557789    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.557975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.557823    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.558061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558010    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:10.558061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558027    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:10.558061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558032    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.558230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558095    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.558230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558218    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.558330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558249    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:10.558330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558271    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token\") pod \"9947727b-4881-419f-a606-55bc58437a3c\" (UID: \"9947727b-4881-419f-a606-55bc58437a3c\") "
Nov 02 15:19:10.558474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558319    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:10.558532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.558465    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:10.559170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.559145    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.559287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.559175    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.559287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.559191    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9947727b-4881-419f-a606-55bc58437a3c-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.559287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.559208    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.559287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.559222    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.560255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.560216    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "kube-api-access-vnbpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:10.560339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.560296    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:10.560520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.560490    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:10.560617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.560543    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token" pod "9947727b-4881-419f-a606-55bc58437a3c" (UID: "9947727b-4881-419f-a606-55bc58437a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:10.660984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.660949    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9947727b-4881-419f-a606-55bc58437a3c-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.660984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.660979    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.660984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.660989    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9947727b-4881-419f-a606-55bc58437a3c-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:10.661207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:10.660998    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbpt\" (UniqueName: \"kubernetes.io/projected/9947727b-4881-419f-a606-55bc58437a3c-kube-api-access-vnbpt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:11.105267 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.105235    5041 generic.go:356] "Generic (PLEG): container finished" podID="9947727b-4881-419f-a606-55bc58437a3c" containerID="0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f" exitCode=0
Nov 02 15:19:11.105490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.105280    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"
Nov 02 15:19:11.105490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.105321    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" event={"ID":"9947727b-4881-419f-a606-55bc58437a3c","Type":"ContainerDied","Data":"0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f"}
Nov 02 15:19:11.105490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.105417    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj" event={"ID":"9947727b-4881-419f-a606-55bc58437a3c","Type":"ContainerDied","Data":"d80ce163728c3bfe79fe843c1af5cd89c43edb6fac35d26664d079cccfcfc3fb"}
Nov 02 15:19:11.105490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.105448    5041 scope.go:122] "RemoveContainer" containerID="0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f"
Nov 02 15:19:11.119623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.119604    5041 scope.go:122] "RemoveContainer" containerID="0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f"
Nov 02 15:19:11.119935 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:11.119911    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f\": container with ID starting with 0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f not found: ID does not exist" containerID="0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f"
Nov 02 15:19:11.120011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.119949    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f"} err="failed to get container status \"0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f\": rpc error: code = NotFound desc = could not find container \"0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f\": container with ID starting with 0ceeddeeab9d7ea51dfd0a526261f7a67d1b86c0b064303c21282fa82d70597f not found: ID does not exist"
Nov 02 15:19:11.155115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.155091    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"]
Nov 02 15:19:11.165082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.165058    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-8l2pj"]
Nov 02 15:19:11.891608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:11.891575    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9947727b-4881-419f-a606-55bc58437a3c" path="/var/lib/kubelet/pods/9947727b-4881-419f-a606-55bc58437a3c/volumes"
Nov 02 15:19:12.964769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:12.964728    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"]
Nov 02 15:19:12.965231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:12.964932    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc" podUID="7b4d69c3-4795-41b3-a81c-4280f002f10d" containerName="maas-api" containerID="cri-o://72a587b24c98a1ae334a578650ed807ba7726c17f28b2975700d60a8d36d94b4" gracePeriod=30
Nov 02 15:19:13.129515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.129483    5041 generic.go:356] "Generic (PLEG): container finished" podID="7b4d69c3-4795-41b3-a81c-4280f002f10d" containerID="72a587b24c98a1ae334a578650ed807ba7726c17f28b2975700d60a8d36d94b4" exitCode=0
Nov 02 15:19:13.129656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.129518    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc" event={"ID":"7b4d69c3-4795-41b3-a81c-4280f002f10d","Type":"ContainerDied","Data":"72a587b24c98a1ae334a578650ed807ba7726c17f28b2975700d60a8d36d94b4"}
Nov 02 15:19:13.464427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.464158    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:19:13.549257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.549205    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6\" (UniqueName: \"kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6\") pod \"7b4d69c3-4795-41b3-a81c-4280f002f10d\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") "
Nov 02 15:19:13.549451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.549352    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs\") pod \"7b4d69c3-4795-41b3-a81c-4280f002f10d\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") "
Nov 02 15:19:13.549544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.549411    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca\") pod \"7b4d69c3-4795-41b3-a81c-4280f002f10d\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") "
Nov 02 15:19:13.552393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.550012    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls\") pod \"7b4d69c3-4795-41b3-a81c-4280f002f10d\" (UID: \"7b4d69c3-4795-41b3-a81c-4280f002f10d\") "
Nov 02 15:19:13.552393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.551189    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca" pod "7b4d69c3-4795-41b3-a81c-4280f002f10d" (UID: "7b4d69c3-4795-41b3-a81c-4280f002f10d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:13.571389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.565727    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d69c3-4795-41b3-a81c-4280f002f10d-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:13.571389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.567686    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs" pod "7b4d69c3-4795-41b3-a81c-4280f002f10d" (UID: "7b4d69c3-4795-41b3-a81c-4280f002f10d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:13.594284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.592187    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls" pod "7b4d69c3-4795-41b3-a81c-4280f002f10d" (UID: "7b4d69c3-4795-41b3-a81c-4280f002f10d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:13.617395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.613848    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6" pod "7b4d69c3-4795-41b3-a81c-4280f002f10d" (UID: "7b4d69c3-4795-41b3-a81c-4280f002f10d"). InnerVolumeSpecName "kube-api-access-gmgq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:13.755388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.754989    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:13.755388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.755035    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgq6\" (UniqueName: \"kubernetes.io/projected/7b4d69c3-4795-41b3-a81c-4280f002f10d-kube-api-access-gmgq6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:13.755388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:13.755052    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d69c3-4795-41b3-a81c-4280f002f10d-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:19:14.168440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:14.168022    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc" event={"ID":"7b4d69c3-4795-41b3-a81c-4280f002f10d","Type":"ContainerDied","Data":"300b43541fb7844142b0d0b02310ffd4b72eb2ec46d4da550ca6db772f5f27cc"}
Nov 02 15:19:14.168440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:14.168101    5041 scope.go:122] "RemoveContainer" containerID="72a587b24c98a1ae334a578650ed807ba7726c17f28b2975700d60a8d36d94b4"
Nov 02 15:19:14.168440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:14.168245    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"
Nov 02 15:19:14.244529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:14.244492    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"]
Nov 02 15:19:14.252989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:14.252955    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7afd1a6-55f6ff8dcf-24zfc"]
Nov 02 15:19:15.891068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:15.891036    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4d69c3-4795-41b3-a81c-4280f002f10d" path="/var/lib/kubelet/pods/7b4d69c3-4795-41b3-a81c-4280f002f10d/volumes"
Nov 02 15:19:29.035331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:29.035296    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"]
Nov 02 15:19:29.044717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:29.044688    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f69f34a-mqd8w"]
Nov 02 15:19:30.073642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:30.073605    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286f1174-4d09-45d2-afbe-96e3c0003ce8" path="/var/lib/kubelet/pods/286f1174-4d09-45d2-afbe-96e3c0003ce8/volumes"
Nov 02 15:19:35.616283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:35.616251    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"]
Nov 02 15:19:35.660344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:35.660300    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"]
Nov 02 15:19:36.779148 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:19:36.779070    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:19:40.616824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:40.616778    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerName="payload-processing" containerID="cri-o://0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047" gracePeriod=25
Nov 02 15:19:40.661214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:40.661168    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerName="payload-pre-processing" containerID="cri-o://126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da" gracePeriod=25
Nov 02 15:19:47.066445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:47.066402    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:19:47.066950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:47.066538    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:19:50.391054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:50.391006    5041 scope.go:122] "RemoveContainer" containerID="ab7eff34b57c665be055e7dd3e236316dcc30bffd49d9ad1c80ecfd77235b484"
Nov 02 15:19:57.065691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:57.065650    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:19:57.066148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:19:57.065651    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:20:00.975325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:00.975292    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:20:01.014090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.014057    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"]
Nov 02 15:20:01.014301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.014276    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j" podUID="5f09b710-9002-47ca-9ebf-20e32057732f" containerName="maas-api" containerID="cri-o://dca4f56b0dea7dccdc5d89bd19be0094e98ef0c911af5d645cc4f92aa90392c8" gracePeriod=30
Nov 02 15:20:01.016324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.016296    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:20:01.055916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.055884    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume\") pod \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") "
Nov 02 15:20:01.057686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.056234    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume\") pod \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") "
Nov 02 15:20:01.057686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.056279    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r\" (UniqueName: \"kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r\") pod \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\" (UID: \"a75fe8c9-9fc6-420d-9ce2-857794e80e16\") "
Nov 02 15:20:01.057686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.056349    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk\") pod \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\" (UID: \"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7\") "
Nov 02 15:20:01.057686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.056381    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume" pod "a75fe8c9-9fc6-420d-9ce2-857794e80e16" (UID: "a75fe8c9-9fc6-420d-9ce2-857794e80e16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:01.057686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.057158    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75fe8c9-9fc6-420d-9ce2-857794e80e16-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:01.057686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.057321    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume" pod "9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" (UID: "9f200b8e-cc5e-4428-a36a-254ec0d1b8d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:01.060586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.060544    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk" pod "9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" (UID: "9f200b8e-cc5e-4428-a36a-254ec0d1b8d7"). InnerVolumeSpecName "kube-api-access-xxjmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:01.060716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.060630    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r" pod "a75fe8c9-9fc6-420d-9ce2-857794e80e16" (UID: "a75fe8c9-9fc6-420d-9ce2-857794e80e16"). InnerVolumeSpecName "kube-api-access-h6q4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:01.158415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.158327    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxjmk\" (UniqueName: \"kubernetes.io/projected/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-kube-api-access-xxjmk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:01.158590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.158576    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:01.158655 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.158648    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6q4r\" (UniqueName: \"kubernetes.io/projected/a75fe8c9-9fc6-420d-9ce2-857794e80e16-kube-api-access-h6q4r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.733898    5041 generic.go:356] "Generic (PLEG): container finished" podID="5f09b710-9002-47ca-9ebf-20e32057732f" containerID="dca4f56b0dea7dccdc5d89bd19be0094e98ef0c911af5d645cc4f92aa90392c8" exitCode=0
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.734150    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j" event={"ID":"5f09b710-9002-47ca-9ebf-20e32057732f","Type":"ContainerDied","Data":"dca4f56b0dea7dccdc5d89bd19be0094e98ef0c911af5d645cc4f92aa90392c8"}
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.736578    5041 generic.go:356] "Generic (PLEG): container finished" podID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerID="0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047" exitCode=0
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.736665    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" event={"ID":"a75fe8c9-9fc6-420d-9ce2-857794e80e16","Type":"ContainerDied","Data":"0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047"}
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.736708    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6" event={"ID":"a75fe8c9-9fc6-420d-9ce2-857794e80e16","Type":"ContainerDied","Data":"6526a4226ca30487e5b3bfc7b1d9c77d3fc4271c0e430d07c6c61389e353ba99"}
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.736736    5041 scope.go:122] "RemoveContainer" containerID="0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047"
Nov 02 15:20:01.741401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.736890    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"
Nov 02 15:20:01.746030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.745991    5041 generic.go:356] "Generic (PLEG): container finished" podID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerID="126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da" exitCode=0
Nov 02 15:20:01.746351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.746329    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" event={"ID":"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7","Type":"ContainerDied","Data":"126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da"}
Nov 02 15:20:01.746526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.746508    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj" event={"ID":"9f200b8e-cc5e-4428-a36a-254ec0d1b8d7","Type":"ContainerDied","Data":"9e420f6ba3388ea8d17a4227060445719bcef893dc4fc1dd796afa74f63a9b66"}
Nov 02 15:20:01.746689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.746676    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"
Nov 02 15:20:01.907824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.871012    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:20:01.955405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.955115    5041 scope.go:122] "RemoveContainer" containerID="0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047"
Nov 02 15:20:01.955654 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:01.955623    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047\": container with ID starting with 0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047 not found: ID does not exist" containerID="0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047"
Nov 02 15:20:01.955776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.955658    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047"} err="failed to get container status \"0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047\": rpc error: code = NotFound desc = could not find container \"0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047\": container with ID starting with 0bb09814e0777618a297bd166677378f0bdd2d0e19933c5694abba913282e047 not found: ID does not exist"
Nov 02 15:20:01.955776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.955679    5041 scope.go:122] "RemoveContainer" containerID="126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da"
Nov 02 15:20:01.974131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.974102    5041 scope.go:122] "RemoveContainer" containerID="126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da"
Nov 02 15:20:01.974578 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:01.974547    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da\": container with ID starting with 126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da not found: ID does not exist" containerID="126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da"
Nov 02 15:20:01.974702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.974591    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da"} err="failed to get container status \"126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da\": rpc error: code = NotFound desc = could not find container \"126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da\": container with ID starting with 126c0bcd108774e61eb2c96ece1a55cd18c2587ba20bf65cba76f47f793a62da not found: ID does not exist"
Nov 02 15:20:01.992937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.991777    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"]
Nov 02 15:20:01.998229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:01.998199    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-4pxcj"]
Nov 02 15:20:02.024448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.024418    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"]
Nov 02 15:20:02.038066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.038032    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-l8gs6"]
Nov 02 15:20:02.071809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.071779    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs\") pod \"5f09b710-9002-47ca-9ebf-20e32057732f\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") "
Nov 02 15:20:02.071972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.071912    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr\" (UniqueName: \"kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr\") pod \"5f09b710-9002-47ca-9ebf-20e32057732f\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") "
Nov 02 15:20:02.072098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.072082    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls\") pod \"5f09b710-9002-47ca-9ebf-20e32057732f\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") "
Nov 02 15:20:02.072171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.072120    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca\") pod \"5f09b710-9002-47ca-9ebf-20e32057732f\" (UID: \"5f09b710-9002-47ca-9ebf-20e32057732f\") "
Nov 02 15:20:02.072624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.072592    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca" pod "5f09b710-9002-47ca-9ebf-20e32057732f" (UID: "5f09b710-9002-47ca-9ebf-20e32057732f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:02.073234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.073200    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f09b710-9002-47ca-9ebf-20e32057732f-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:02.074423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.074394    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr" pod "5f09b710-9002-47ca-9ebf-20e32057732f" (UID: "5f09b710-9002-47ca-9ebf-20e32057732f"). InnerVolumeSpecName "kube-api-access-9tsnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:02.074622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.074590    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs" pod "5f09b710-9002-47ca-9ebf-20e32057732f" (UID: "5f09b710-9002-47ca-9ebf-20e32057732f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:02.074719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.074615    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls" pod "5f09b710-9002-47ca-9ebf-20e32057732f" (UID: "5f09b710-9002-47ca-9ebf-20e32057732f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:02.188440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.175397    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:02.188440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.175446    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f09b710-9002-47ca-9ebf-20e32057732f-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:02.188440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.175462    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tsnr\" (UniqueName: \"kubernetes.io/projected/5f09b710-9002-47ca-9ebf-20e32057732f-kube-api-access-9tsnr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:02.756257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.756226    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"
Nov 02 15:20:02.756257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.756245    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j" event={"ID":"5f09b710-9002-47ca-9ebf-20e32057732f","Type":"ContainerDied","Data":"aa479d2439a68a090ce77c14e7a9bfe30beda418e8a15ca434192436dfa89076"}
Nov 02 15:20:02.756559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.756311    5041 scope.go:122] "RemoveContainer" containerID="dca4f56b0dea7dccdc5d89bd19be0094e98ef0c911af5d645cc4f92aa90392c8"
Nov 02 15:20:02.805096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.805067    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"]
Nov 02 15:20:02.814390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:02.814351    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad276f-597478b7f-29q7j"]
Nov 02 15:20:03.890496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:03.890464    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f09b710-9002-47ca-9ebf-20e32057732f" path="/var/lib/kubelet/pods/5f09b710-9002-47ca-9ebf-20e32057732f/volumes"
Nov 02 15:20:03.890969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:03.890954    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" path="/var/lib/kubelet/pods/9f200b8e-cc5e-4428-a36a-254ec0d1b8d7/volumes"
Nov 02 15:20:03.891346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:03.891333    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" path="/var/lib/kubelet/pods/a75fe8c9-9fc6-420d-9ce2-857794e80e16/volumes"
Nov 02 15:20:06.770617 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:06.770582    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:20:14.552026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.551992    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"]
Nov 02 15:20:14.552916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552898    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552918    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552929    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552935    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552948    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerName="payload-processing"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552953    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerName="payload-processing"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552964    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552969    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552976    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f09b710-9002-47ca-9ebf-20e32057732f" containerName="maas-api"
Nov 02 15:20:14.552979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552980    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f09b710-9002-47ca-9ebf-20e32057732f" containerName="maas-api"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552992    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ab3f400f-be12-41bc-86a6-f0a1723ea179" containerName="payload-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.552996    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3f400f-be12-41bc-86a6-f0a1723ea179" containerName="payload-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553006    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7b4d69c3-4795-41b3-a81c-4280f002f10d" containerName="maas-api"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553011    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4d69c3-4795-41b3-a81c-4280f002f10d" containerName="maas-api"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553020    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553024    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553043    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" containerName="payload-pre-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553048    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" containerName="payload-pre-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553063    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerName="payload-pre-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553067    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerName="payload-pre-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553090    5041 cpu_manager.go:414] "containerMap: removing container" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553094    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553216    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3f400f-be12-41bc-86a6-f0a1723ea179" containerName="payload-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553226    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4d69c3-4795-41b3-a81c-4280f002f10d" containerName="maas-api"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553232    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3c9823-4ec5-443a-86a2-dba33a1921cd" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553242    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c8f6d6-8f77-4b25-a3ed-2b50602b3719" containerName="payload-pre-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553248    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f200b8e-cc5e-4428-a36a-254ec0d1b8d7" containerName="payload-pre-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553256    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a6879a-f491-4692-82c5-6474fafae08d" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553264    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e259152-910d-43c6-a8fb-5b231bb2632c" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553269    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9947727b-4881-419f-a606-55bc58437a3c" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553274    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75fe8c9-9fc6-420d-9ce2-857794e80e16" containerName="payload-processing"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553280    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3604113c-a16f-42b0-b65e-f0e3456c6156" containerName="istio-proxy"
Nov 02 15:20:14.553275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.553286    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f09b710-9002-47ca-9ebf-20e32057732f" containerName="maas-api"
Nov 02 15:20:14.557453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.557434    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.573055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.573028    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-78d075-serving-cert\""
Nov 02 15:20:14.608474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.608444    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"]
Nov 02 15:20:14.650677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.650640    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.651005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.650974    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.651152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.651085    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.651261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.651243    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.780406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.759513    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.780775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.780742    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.780940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.780922    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.781317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.781297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.804431 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:14.782244    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-serving-cert: secret "maas-api-e2e-shared-b-w4-78d075-serving-cert" not found
Nov 02 15:20:14.804706 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:14.804681    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls podName:63bff263-0915-44d7-92e1-781e193fdf12 nodeName:}" failed. No retries permitted until 2026-11-02 15:20:15.304655618 +0000 UTC m=+1467.562725065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls") pod "maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" (UID: "63bff263-0915-44d7-92e1-781e193fdf12") : secret "maas-api-e2e-shared-b-w4-78d075-serving-cert" not found
Nov 02 15:20:14.804925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.800572    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.805032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.782984    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:14.840401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:14.837078    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:15.307392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:15.307347    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:15.309871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:15.309844    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:15.499391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:15.495734    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:16.500560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:16.500527    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"]
Nov 02 15:20:16.500962 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:16.500900    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bff263_0915_44d7_92e1_781e193fdf12.slice/crio-b9ba5962a96d21d77779a1ea0c080ce73e91e2d0b693836f929e5cbbfede8bca WatchSource:0}: Error finding container b9ba5962a96d21d77779a1ea0c080ce73e91e2d0b693836f929e5cbbfede8bca: Status 404 returned error can't find the container with id b9ba5962a96d21d77779a1ea0c080ce73e91e2d0b693836f929e5cbbfede8bca
Nov 02 15:20:16.502569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:16.502549    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:20:17.208749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:17.208703    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" event={"ID":"63bff263-0915-44d7-92e1-781e193fdf12","Type":"ContainerStarted","Data":"4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c"}
Nov 02 15:20:17.208941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:17.208847    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:17.208941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:17.208875    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" event={"ID":"63bff263-0915-44d7-92e1-781e193fdf12","Type":"ContainerStarted","Data":"b9ba5962a96d21d77779a1ea0c080ce73e91e2d0b693836f929e5cbbfede8bca"}
Nov 02 15:20:17.265319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:17.265268    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" podStartSLOduration=2.88795058 podStartE2EDuration="3.265253523s" totalImagesPullingTime="377.302943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:17.262776908 +0000 UTC m=+1469.520846352" watchObservedRunningTime="2026-11-02 15:20:17.265253523 +0000 UTC m=+1469.523322975"
Nov 02 15:20:23.227250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:23.227216    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:20:27.564729 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:27.564694    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"]
Nov 02 15:20:27.565192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:27.564912    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api" containerID="cri-o://8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20" gracePeriod=30
Nov 02 15:20:28.342437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.342224    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:20:28.468386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.466991    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\") pod \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") "
Nov 02 15:20:28.468386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.467104    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs\") pod \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") "
Nov 02 15:20:28.468386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.467140    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca\") pod \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") "
Nov 02 15:20:28.468386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.467309    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2\" (UniqueName: \"kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2\") pod \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\" (UID: \"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2\") "
Nov 02 15:20:28.473997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.473955    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca" pod "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" (UID: "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:28.482483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.480501    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs" pod "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" (UID: "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:28.482483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.480796    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls" pod "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" (UID: "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:28.482791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.482741    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2" pod "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" (UID: "b26f0bf2-f8f7-416d-ba83-2c4b58c742b2"). InnerVolumeSpecName "kube-api-access-llgp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:28.484535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.484504    5041 generic.go:356] "Generic (PLEG): container finished" podID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerID="8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20" exitCode=0
Nov 02 15:20:28.484638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.484623    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerDied","Data":"8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20"}
Nov 02 15:20:28.484704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.484677    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm" event={"ID":"b26f0bf2-f8f7-416d-ba83-2c4b58c742b2","Type":"ContainerDied","Data":"7bb691c5aee37d8b9cc5b829213f964b4acd0ef423d721ca07af108f4d8d107c"}
Nov 02 15:20:28.484765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.484710    5041 scope.go:122] "RemoveContainer" containerID="8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20"
Nov 02 15:20:28.484912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.484891    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"
Nov 02 15:20:28.512899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.512877    5041 scope.go:122] "RemoveContainer" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"
Nov 02 15:20:28.526478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.526449    5041 scope.go:122] "RemoveContainer" containerID="8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20"
Nov 02 15:20:28.527002 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:28.526979    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20\": container with ID starting with 8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20 not found: ID does not exist" containerID="8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20"
Nov 02 15:20:28.527105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.527011    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20"} err="failed to get container status \"8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20\": rpc error: code = NotFound desc = could not find container \"8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20\": container with ID starting with 8e38fab6575409d1dffc300278ef85ab54210846c56b0801bc99c10e5af3aa20 not found: ID does not exist"
Nov 02 15:20:28.527105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.527030    5041 scope.go:122] "RemoveContainer" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"
Nov 02 15:20:28.527408 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:28.527343    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53\": container with ID starting with cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53 not found: ID does not exist" containerID="cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"
Nov 02 15:20:28.527523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.527417    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53"} err="failed to get container status \"cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53\": rpc error: code = NotFound desc = could not find container \"cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53\": container with ID starting with cad2227983b8548999d53778a8ec849f8837a15850c29a77ad5ebe86d9c2bd53 not found: ID does not exist"
Nov 02 15:20:28.542727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.542702    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"]
Nov 02 15:20:28.552484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.552465    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-hvpsm"]
Nov 02 15:20:28.569452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.569427    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:28.569452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.569451    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:28.569877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.569463    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgp2\" (UniqueName: \"kubernetes.io/projected/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-kube-api-access-llgp2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:28.569877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:28.569472    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:20:29.891484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:29.891450    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" path="/var/lib/kubelet/pods/b26f0bf2-f8f7-416d-ba83-2c4b58c742b2/volumes"
Nov 02 15:20:32.291201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.291165    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"]
Nov 02 15:20:32.292099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292079    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292100    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292120    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292127    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292145    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292151    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292275    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292288    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.292335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.292296    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:32.297895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.297877    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.313459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.313436    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2bbcb5-serving-cert\""
Nov 02 15:20:32.323540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.323511    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"]
Nov 02 15:20:32.369249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.369221    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwd58\" (UniqueName: \"kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.369415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.369313    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.369535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.369517    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.369595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.369572    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.472460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.471065    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.472460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.471280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.472460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.471380    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.472460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.471593    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwd58\" (UniqueName: \"kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.473114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.472905    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.495188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.495120    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.504065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.504004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.514540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.514474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwd58\" (UniqueName: \"kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58\") pod \"maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.621623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.621541    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:32.936730 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:32.936699    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcea4fe7_a375_4ffb_8a2d_8964ca4795ee.slice/crio-3d0cd34886baf85696d46c8898d189378b8ae6562216454dfe54b940386d38eb WatchSource:0}: Error finding container 3d0cd34886baf85696d46c8898d189378b8ae6562216454dfe54b940386d38eb: Status 404 returned error can't find the container with id 3d0cd34886baf85696d46c8898d189378b8ae6562216454dfe54b940386d38eb
Nov 02 15:20:32.937012 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:32.936991    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"]
Nov 02 15:20:33.611985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:33.611941    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4" event={"ID":"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee","Type":"ContainerStarted","Data":"bb8bd8d697fbcd15f688bdf2b9dc9af84581712b9a1a1e8b60a009b33e576d88"}
Nov 02 15:20:33.612499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:33.612013    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4" event={"ID":"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee","Type":"ContainerStarted","Data":"3d0cd34886baf85696d46c8898d189378b8ae6562216454dfe54b940386d38eb"}
Nov 02 15:20:33.612499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:33.612043    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:33.658634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:33.658402    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4" podStartSLOduration=1.294575717 podStartE2EDuration="1.658373951s" totalImagesPullingTime="363.798234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:33.654510729 +0000 UTC m=+1485.912580179" watchObservedRunningTime="2026-11-02 15:20:33.658373951 +0000 UTC m=+1485.916443393"
Nov 02 15:20:36.844391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.842654    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"]
Nov 02 15:20:36.844391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.844047    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:36.844391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.844074    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:36.845099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.844401    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26f0bf2-f8f7-416d-ba83-2c4b58c742b2" containerName="maas-api"
Nov 02 15:20:36.867397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.864001    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:36.901044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.899611    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-db75ff-dockercfg-9dm54\""
Nov 02 15:20:36.971432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:36.939848    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-db75ff\""
Nov 02 15:20:37.149880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.135077    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxz9\" (UniqueName: \"kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9\") pod \"payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.150261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.150227    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.196401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.185393    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"]
Nov 02 15:20:37.272385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.266822    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxz9\" (UniqueName: \"kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9\") pod \"payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.272385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.266918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.272385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.267947    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.301582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.292954    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"]
Nov 02 15:20:37.310239 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:20:37.309416    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:20:37.311946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.311815    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"]
Nov 02 15:20:37.312139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.311914    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.332765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.332744    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxz9\" (UniqueName: \"kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9\") pod \"payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.470379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.470331    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") " pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.470650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.470627    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppvl\" (UniqueName: \"kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl\") pod \"payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") " pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.481137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.481102    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:37.574449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.574404    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") " pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.575353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.575324    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppvl\" (UniqueName: \"kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl\") pod \"payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") " pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.576701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.576677    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") " pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.600858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.600813    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppvl\" (UniqueName: \"kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl\") pod \"payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") " pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.633527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.633498    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:37.798512 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:37.798476    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ceb140_f94a_46f0_a34d_54b9e542ccdb.slice/crio-68424e918a0d99fa7e11a941e76e26fd9e4844b64db0a7e3a7c3d382fd68a0a4 WatchSource:0}: Error finding container 68424e918a0d99fa7e11a941e76e26fd9e4844b64db0a7e3a7c3d382fd68a0a4: Status 404 returned error can't find the container with id 68424e918a0d99fa7e11a941e76e26fd9e4844b64db0a7e3a7c3d382fd68a0a4
Nov 02 15:20:37.801599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.801572    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"]
Nov 02 15:20:37.952029 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:37.952000    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23103073_28d9_454e_b080_3ce8a4caaae9.slice/crio-31bea0217ccdc55243b37db3937636c850b97f569c6e9f7dc26e92de8aa35821 WatchSource:0}: Error finding container 31bea0217ccdc55243b37db3937636c850b97f569c6e9f7dc26e92de8aa35821: Status 404 returned error can't find the container with id 31bea0217ccdc55243b37db3937636c850b97f569c6e9f7dc26e92de8aa35821
Nov 02 15:20:37.952646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:37.952595    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"]
Nov 02 15:20:38.663670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.663629    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg" event={"ID":"96ceb140-f94a-46f0-a34d-54b9e542ccdb","Type":"ContainerStarted","Data":"09abd27254c60c98890628cf6737bb5a12069c3f0ec03a5b3e6f657e1badbb41"}
Nov 02 15:20:38.663864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.663736    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:38.663864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.663760    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg" event={"ID":"96ceb140-f94a-46f0-a34d-54b9e542ccdb","Type":"ContainerStarted","Data":"68424e918a0d99fa7e11a941e76e26fd9e4844b64db0a7e3a7c3d382fd68a0a4"}
Nov 02 15:20:38.665657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.665628    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm" event={"ID":"23103073-28d9-454e-b080-3ce8a4caaae9","Type":"ContainerStarted","Data":"4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59"}
Nov 02 15:20:38.665785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.665685    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm" event={"ID":"23103073-28d9-454e-b080-3ce8a4caaae9","Type":"ContainerStarted","Data":"31bea0217ccdc55243b37db3937636c850b97f569c6e9f7dc26e92de8aa35821"}
Nov 02 15:20:38.665785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.665712    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:38.711859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.711809    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg" podStartSLOduration=2.711797294 podStartE2EDuration="2.711797294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:38.708007468 +0000 UTC m=+1490.966076920" watchObservedRunningTime="2026-11-02 15:20:38.711797294 +0000 UTC m=+1490.969866743"
Nov 02 15:20:38.722674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.722643    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"]
Nov 02 15:20:38.727586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.727567    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:38.741523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.741499    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3d7df16e-dockercfg-lrhpp\""
Nov 02 15:20:38.741523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.741498    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d7df16e\""
Nov 02 15:20:38.759469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.759444    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"]
Nov 02 15:20:38.760829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.760789    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm" podStartSLOduration=2.760776629 podStartE2EDuration="2.760776629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:38.753622363 +0000 UTC m=+1491.011691815" watchObservedRunningTime="2026-11-02 15:20:38.760776629 +0000 UTC m=+1491.018846446"
Nov 02 15:20:38.832727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.832693    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khc8l\" (UniqueName: \"kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:38.832877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.832813    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:38.852205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.852177    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"]
Nov 02 15:20:38.858266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.858245    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:38.882401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.881582    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"]
Nov 02 15:20:38.954388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.947187    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:38.954388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.947328    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcnt\" (UniqueName: \"kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:38.994879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:38.994563    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:39.071972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.062611    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khc8l\" (UniqueName: \"kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:39.071972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.062687    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:39.129946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.121881    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khc8l\" (UniqueName: \"kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l\") pod \"payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:39.182211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.182168    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcnt\" (UniqueName: \"kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:39.199399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.182873    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:39.200983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.198571    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:39.291663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.291354    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcnt\" (UniqueName: \"kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt\") pod \"payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") " pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:39.347017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.346986    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:39.476325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.476296    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:39.631484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.631459    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:20:39.687032 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:39.687000    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef371da_31d2_436e_a31b_27408bc91870.slice/crio-6c3aae97a954b77dfdfda7633a7bd4f1592ccf40ba6d8061509106c574073c88 WatchSource:0}: Error finding container 6c3aae97a954b77dfdfda7633a7bd4f1592ccf40ba6d8061509106c574073c88: Status 404 returned error can't find the container with id 6c3aae97a954b77dfdfda7633a7bd4f1592ccf40ba6d8061509106c574073c88
Nov 02 15:20:39.687153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.687089    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"]
Nov 02 15:20:39.795607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:39.795536    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"]
Nov 02 15:20:39.801316 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:39.801286    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5747cbe_3566_4b3a_83cd_8a2400e832d4.slice/crio-2b34c0f2d43f178ea648659c36271fde6e002925c24a27f70e7e81220184040b WatchSource:0}: Error finding container 2b34c0f2d43f178ea648659c36271fde6e002925c24a27f70e7e81220184040b: Status 404 returned error can't find the container with id 2b34c0f2d43f178ea648659c36271fde6e002925c24a27f70e7e81220184040b
Nov 02 15:20:40.687856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.687821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq" event={"ID":"5ef371da-31d2-436e-a31b-27408bc91870","Type":"ContainerStarted","Data":"9641c012ccf0544e4ed75a925492c5e0798e7cec75383e0ddf3be3efbcd76f1f"}
Nov 02 15:20:40.688321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.687879    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq" event={"ID":"5ef371da-31d2-436e-a31b-27408bc91870","Type":"ContainerStarted","Data":"6c3aae97a954b77dfdfda7633a7bd4f1592ccf40ba6d8061509106c574073c88"}
Nov 02 15:20:40.688321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.687908    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:40.689934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.689902    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s" event={"ID":"c5747cbe-3566-4b3a-83cd-8a2400e832d4","Type":"ContainerStarted","Data":"837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603"}
Nov 02 15:20:40.690049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.689960    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s" event={"ID":"c5747cbe-3566-4b3a-83cd-8a2400e832d4","Type":"ContainerStarted","Data":"2b34c0f2d43f178ea648659c36271fde6e002925c24a27f70e7e81220184040b"}
Nov 02 15:20:40.690211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.690194    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:40.727640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.727592    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq" podStartSLOduration=2.7275791700000003 podStartE2EDuration="2.72757917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:40.724399717 +0000 UTC m=+1492.982469168" watchObservedRunningTime="2026-11-02 15:20:40.72757917 +0000 UTC m=+1492.985648620"
Nov 02 15:20:40.853192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:40.853130    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s" podStartSLOduration=2.853108495 podStartE2EDuration="2.853108495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:40.782789239 +0000 UTC m=+1493.040858689" watchObservedRunningTime="2026-11-02 15:20:40.853108495 +0000 UTC m=+1493.111177948"
Nov 02 15:20:48.854331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:48.854254    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:20:48.857642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:48.856220    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:20:49.717385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:49.712597    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:20:49.717385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:49.713985    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:20:50.487880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:50.487846    5041 scope.go:122] "RemoveContainer" containerID="8707500bd71d81707ad382b75c80902ed4458d1d041dff3569c0d48dadd9d684"
Nov 02 15:20:50.499414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:50.499387    5041 scope.go:122] "RemoveContainer" containerID="35b82db8f048c559e56e2645e7d90302680458b52e529b4fd2a284fa1943d9d1"
Nov 02 15:20:51.180582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.180544    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"]
Nov 02 15:20:51.192066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.192028    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.193015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.192967    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"]
Nov 02 15:20:51.208347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.208326    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-815746\""
Nov 02 15:20:51.218193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.218165    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-815746-dockercfg-g64p5\""
Nov 02 15:20:51.346425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.346391    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-815746-78756d469f-lcxkm\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.346586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.346565    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwqf\" (UniqueName: \"kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf\") pod \"payload-processing-e2e-api-b-815746-78756d469f-lcxkm\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.448213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.448147    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-815746-78756d469f-lcxkm\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.448333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.448302    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwqf\" (UniqueName: \"kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf\") pod \"payload-processing-e2e-api-b-815746-78756d469f-lcxkm\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.448900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.448882    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-815746-78756d469f-lcxkm\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.469249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.469224    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwqf\" (UniqueName: \"kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf\") pod \"payload-processing-e2e-api-b-815746-78756d469f-lcxkm\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.509099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.509075    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:51.776442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.775421    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:20:51.776442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.776224    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:20:51.825808 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:51.825051    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4783afee_ed79_4776_ab97_289b6f26c7ae.slice/crio-e67a7658d21671d560e2cfa62cb7185965e1ffd80a44821d26edce4d7f4f6a2d WatchSource:0}: Error finding container e67a7658d21671d560e2cfa62cb7185965e1ffd80a44821d26edce4d7f4f6a2d: Status 404 returned error can't find the container with id e67a7658d21671d560e2cfa62cb7185965e1ffd80a44821d26edce4d7f4f6a2d
Nov 02 15:20:51.825808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:51.825483    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"]
Nov 02 15:20:52.180128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:52.180081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm" event={"ID":"4783afee-ed79-4776-ab97-289b6f26c7ae","Type":"ContainerStarted","Data":"1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f"}
Nov 02 15:20:52.180292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:52.180151    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm" event={"ID":"4783afee-ed79-4776-ab97-289b6f26c7ae","Type":"ContainerStarted","Data":"e67a7658d21671d560e2cfa62cb7185965e1ffd80a44821d26edce4d7f4f6a2d"}
Nov 02 15:20:52.180292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:52.180185    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:20:52.224967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:52.224919    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm" podStartSLOduration=2.224906378 podStartE2EDuration="2.224906378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:52.219279485 +0000 UTC m=+1504.477348934" watchObservedRunningTime="2026-11-02 15:20:52.224906378 +0000 UTC m=+1504.482975828"
Nov 02 15:20:58.257766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.257725    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"]
Nov 02 15:20:58.267614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.267588    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.287015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.286987    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1cdbe6-serving-cert\""
Nov 02 15:20:58.287591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.287563    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"]
Nov 02 15:20:58.323345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.323315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.323560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.323503    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmw4j\" (UniqueName: \"kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.323622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.323592    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.323680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.323634    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.425162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.425129    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.425332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.425214    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmw4j\" (UniqueName: \"kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.425332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.425249    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.425332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.425288    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.425965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.425931    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.427920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.427892    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.428077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.427903    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.477805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.477771    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmw4j\" (UniqueName: \"kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j\") pod \"maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.588469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.585873    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:20:58.902525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:58.902498    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"]
Nov 02 15:20:58.902740 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:20:58.902717    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ac4dee_61fe_4c9d_8e2e_09ba70976487.slice/crio-b058d2fb40031c029ac399745ca06d8d4be1c51d99422cbd8c7e2f222caebf88 WatchSource:0}: Error finding container b058d2fb40031c029ac399745ca06d8d4be1c51d99422cbd8c7e2f222caebf88: Status 404 returned error can't find the container with id b058d2fb40031c029ac399745ca06d8d4be1c51d99422cbd8c7e2f222caebf88
Nov 02 15:20:59.284552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:20:59.284525    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" event={"ID":"45ac4dee-61fe-4c9d-8e2e-09ba70976487","Type":"ContainerStarted","Data":"b058d2fb40031c029ac399745ca06d8d4be1c51d99422cbd8c7e2f222caebf88"}
Nov 02 15:21:00.295490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:00.295456    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" event={"ID":"45ac4dee-61fe-4c9d-8e2e-09ba70976487","Type":"ContainerStarted","Data":"ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"}
Nov 02 15:21:00.295910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:00.295685    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:21:00.336177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:00.336130    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" podStartSLOduration=2.011848232 podStartE2EDuration="2.336116839s" totalImagesPullingTime="324.268607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:00.334536316 +0000 UTC m=+1512.592605768" watchObservedRunningTime="2026-11-02 15:21:00.336116839 +0000 UTC m=+1512.594186290"
Nov 02 15:21:03.241387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:03.238561    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:21:06.306234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:06.306193    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.221:8443/health\": dial tcp 10.132.0.221:8443: connect: connection refused"
Nov 02 15:21:06.841835 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:21:06.841796    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:21:09.601182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:09.601096    5041 generic.go:356] "Generic (PLEG): container finished" podID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerID="ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8" exitCode=1
Nov 02 15:21:09.601182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:09.601161    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" event={"ID":"45ac4dee-61fe-4c9d-8e2e-09ba70976487","Type":"ContainerDied","Data":"ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"}
Nov 02 15:21:09.601638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:09.601564    5041 scope.go:122] "RemoveContainer" containerID="ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"
Nov 02 15:21:10.613218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:10.613183    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" event={"ID":"45ac4dee-61fe-4c9d-8e2e-09ba70976487","Type":"ContainerStarted","Data":"502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39"}
Nov 02 15:21:10.613683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:10.613422    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:21:16.628647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:16.628618    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:21:28.038407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.038359    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"]
Nov 02 15:21:28.045040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.045016    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.071103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.064561    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-815746-serving-cert\""
Nov 02 15:21:28.071103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.071024    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"]
Nov 02 15:21:28.192896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.192854    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv6c\" (UniqueName: \"kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.193162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.193134    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.193297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.193281    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.193440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.193422    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.297626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.297504    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.298787 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.298743    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.301873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.301849    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.302561 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.302540    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv6c\" (UniqueName: \"kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.303345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.301689    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.321470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.317055    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.328345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.328282    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.334857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.334794    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv6c\" (UniqueName: \"kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c\") pod \"maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.364681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.364298    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:28.683829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:28.683691    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"]
Nov 02 15:21:29.015773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:29.015734    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz" event={"ID":"646b75bc-7edd-46cb-bc61-125e95b3598a","Type":"ContainerStarted","Data":"5dab8d5b20a80dc415f043090f6e9c1fe5a2b7a30d549b875ea17b0541b44da8"}
Nov 02 15:21:30.042221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:30.042112    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz" event={"ID":"646b75bc-7edd-46cb-bc61-125e95b3598a","Type":"ContainerStarted","Data":"dd4188ccc5eddedb9af047b90e672d2a6bfbb0025000453872e39e1dc9264529"}
Nov 02 15:21:30.042221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:30.042189    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:30.087489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:30.087433    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz" podStartSLOduration=1.7704566480000001 podStartE2EDuration="2.087416861s" totalImagesPullingTime="316.960213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:30.081443828 +0000 UTC m=+1542.339513279" watchObservedRunningTime="2026-11-02 15:21:30.087416861 +0000 UTC m=+1542.345486311"
Nov 02 15:21:36.056962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:21:36.056927    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:21:36.770802 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:21:36.770763    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:22:02.428418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.428324    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"]
Nov 02 15:22:02.433924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.433900    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.450432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.449389    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f975ed-serving-cert\""
Nov 02 15:22:02.475349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.475319    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"]
Nov 02 15:22:02.489952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.489911    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qb9\" (UniqueName: \"kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.490115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.490049    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.490115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.490124    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.490327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.490187    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.591210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.591175    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.591412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.591268    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.591412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.591396    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qb9\" (UniqueName: \"kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.591544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.591500    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.591933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.591903    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.593643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.593621    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.593790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.593772    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.612470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.612445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qb9\" (UniqueName: \"kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9\") pod \"maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:02.781450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:02.780671    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:03.278237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:03.278195    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"]
Nov 02 15:22:04.277437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:04.276507    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b" event={"ID":"c4af444f-93fa-4ea2-a652-fab7df58fbc2","Type":"ContainerStarted","Data":"cf1468cd0c4c01b8e013c44571480fdcf22581ad9b13e5f9d9b6725ebd0b2d94"}
Nov 02 15:22:04.277437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:04.276601    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b" event={"ID":"c4af444f-93fa-4ea2-a652-fab7df58fbc2","Type":"ContainerStarted","Data":"02da6d3d8499b94b60df44a6594232cf7e321b26021d635ba1bb395a2c642d92"}
Nov 02 15:22:04.277963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:04.277672    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:04.321553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:04.321504    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b" podStartSLOduration=1.831121979 podStartE2EDuration="2.321488742s" totalImagesPullingTime="490.366763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:04.318313179 +0000 UTC m=+1576.576382629" watchObservedRunningTime="2026-11-02 15:22:04.321488742 +0000 UTC m=+1576.579558192"
Nov 02 15:22:06.776016 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:06.775981    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:22:11.106992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:11.106944    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"]
Nov 02 15:22:11.107646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:11.107205    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy" containerID="cri-o://657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73" gracePeriod=30
Nov 02 15:22:11.302434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:11.302403    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:22:11.522866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:11.522832    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"]
Nov 02 15:22:11.523070 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:11.523044    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy" containerID="cri-o://f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404" gracePeriod=30
Nov 02 15:22:14.051636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:14.051601    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"]
Nov 02 15:22:14.091552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:14.091504    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"]
Nov 02 15:22:16.455143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.455119    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:22:16.490294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.490260    5041 generic.go:356] "Generic (PLEG): container finished" podID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerID="657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73" exitCode=0
Nov 02 15:22:16.490463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.490299    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"
Nov 02 15:22:16.490463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.490327    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" event={"ID":"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd","Type":"ContainerDied","Data":"657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73"}
Nov 02 15:22:16.490463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.490390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68" event={"ID":"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd","Type":"ContainerDied","Data":"165c304c57fbb91ac68b23e575833dd5cdb19e161460ba20202540f5fed3f278"}
Nov 02 15:22:16.490463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.490412    5041 scope.go:122] "RemoveContainer" containerID="657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73"
Nov 02 15:22:16.501765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.501748    5041 scope.go:122] "RemoveContainer" containerID="657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73"
Nov 02 15:22:16.502047 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:16.502023    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73\": container with ID starting with 657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73 not found: ID does not exist" containerID="657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73"
Nov 02 15:22:16.502129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.502058    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73"} err="failed to get container status \"657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73\": rpc error: code = NotFound desc = could not find container \"657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73\": container with ID starting with 657ad152235a9bec25d106fd36e5e8eb55fab4be09012f7dd2b858cb9aefcd73 not found: ID does not exist"
Nov 02 15:22:16.549895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.549867    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.549924    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.549970    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550071    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550148    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550217    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550207    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.550555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550529    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.550613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550565    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:16.550674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.550617    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token\") pod \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\" (UID: \"8a5665b1-cf1b-42f4-8a2a-d8ac829020cd\") "
Nov 02 15:22:16.551428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.551402    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.551582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.551431    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.552428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.552234    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.552633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.552582    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.552974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.552953    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.553180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.553153    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:16.554115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.554090    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:16.554263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.554243    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "kube-api-access-8dqdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:16.554983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.554956    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy" pod "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" (UID: "8a5665b1-cf1b-42f4-8a2a-d8ac829020cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.656624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656599    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.656684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656629    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dqdd\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-kube-api-access-8dqdd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.656684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656643    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.656684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656658    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.656684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656671    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.656826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656685    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.656826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.656699    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.770597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.770563    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"]
Nov 02 15:22:16.770872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.770844    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy" containerID="cri-o://b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035" gracePeriod=30
Nov 02 15:22:16.831097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.831072    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:22:16.859017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.858959    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859042    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859128    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859169    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859243    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859437    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859619    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859696    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859829    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data\") pod \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\" (UID: \"2b1fd59e-e864-4bc6-934e-af5327b9f84f\") "
Nov 02 15:22:16.859903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.859841    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.862427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.862357    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:16.862596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.862575    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"]
Nov 02 15:22:16.862957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.862934    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.863080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.863053    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.863166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.863139    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:16.863509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.863482    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.865113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.865067    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "kube-api-access-7knfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:16.865265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.865185    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.865265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.865201    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.865265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.865214    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.865508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.865482    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:16.865766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.865749    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy" pod "2b1fd59e-e864-4bc6-934e-af5327b9f84f" (UID: "2b1fd59e-e864-4bc6-934e-af5327b9f84f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:16.866862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.866841    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-lsn68"]
Nov 02 15:22:16.966960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.966926    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.966960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.966961    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.967218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.966977    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.967218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.966991    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7knfk\" (UniqueName: \"kubernetes.io/projected/2b1fd59e-e864-4bc6-934e-af5327b9f84f-kube-api-access-7knfk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.967218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.967005    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:16.967218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:16.967019    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1fd59e-e864-4bc6-934e-af5327b9f84f-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.502639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.502606    5041 generic.go:356] "Generic (PLEG): container finished" podID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerID="f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404" exitCode=0
Nov 02 15:22:17.503054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.502683    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"
Nov 02 15:22:17.503054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.502688    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" event={"ID":"2b1fd59e-e864-4bc6-934e-af5327b9f84f","Type":"ContainerDied","Data":"f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404"}
Nov 02 15:22:17.503054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.502790    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl" event={"ID":"2b1fd59e-e864-4bc6-934e-af5327b9f84f","Type":"ContainerDied","Data":"40acf5d5a07a6fd6e506cb873ed8c311ac324e2b0bb9d28cc727346e9d1c2fdf"}
Nov 02 15:22:17.503054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.502809    5041 scope.go:122] "RemoveContainer" containerID="f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404"
Nov 02 15:22:17.516484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.516463    5041 scope.go:122] "RemoveContainer" containerID="f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404"
Nov 02 15:22:17.516750 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:17.516732    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404\": container with ID starting with f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404 not found: ID does not exist" containerID="f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404"
Nov 02 15:22:17.516818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.516757    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404"} err="failed to get container status \"f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404\": rpc error: code = NotFound desc = could not find container \"f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404\": container with ID starting with f40e86464db73a1949126a8fe3caeb85058476c396b746c93115c7db992be404 not found: ID does not exist"
Nov 02 15:22:17.554119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.554082    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"]
Nov 02 15:22:17.560278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.560252    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-8mjkl"]
Nov 02 15:22:17.988386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.973808    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" path="/var/lib/kubelet/pods/2b1fd59e-e864-4bc6-934e-af5327b9f84f/volumes"
Nov 02 15:22:17.988386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:17.976973    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" path="/var/lib/kubelet/pods/8a5665b1-cf1b-42f4-8a2a-d8ac829020cd/volumes"
Nov 02 15:22:19.052826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.052773    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s" podUID="c5747cbe-3566-4b3a-83cd-8a2400e832d4" containerName="payload-processing" containerID="cri-o://837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603" gracePeriod=25
Nov 02 15:22:19.369509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.369474    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:22:19.399493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.399456    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq" podUID="5ef371da-31d2-436e-a31b-27408bc91870" containerName="payload-pre-processing" containerID="cri-o://9641c012ccf0544e4ed75a925492c5e0798e7cec75383e0ddf3be3efbcd76f1f" gracePeriod=25
Nov 02 15:22:19.428440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.427651    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume\") pod \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") "
Nov 02 15:22:19.428440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.427739    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt\" (UniqueName: \"kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt\") pod \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\" (UID: \"c5747cbe-3566-4b3a-83cd-8a2400e832d4\") "
Nov 02 15:22:19.429487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.429460    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume" pod "c5747cbe-3566-4b3a-83cd-8a2400e832d4" (UID: "c5747cbe-3566-4b3a-83cd-8a2400e832d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:19.431645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.431616    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt" pod "c5747cbe-3566-4b3a-83cd-8a2400e832d4" (UID: "c5747cbe-3566-4b3a-83cd-8a2400e832d4"). InnerVolumeSpecName "kube-api-access-tvcnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:19.524869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.524839    5041 generic.go:356] "Generic (PLEG): container finished" podID="5ef371da-31d2-436e-a31b-27408bc91870" containerID="9641c012ccf0544e4ed75a925492c5e0798e7cec75383e0ddf3be3efbcd76f1f" exitCode=0
Nov 02 15:22:19.524978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.524908    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq" event={"ID":"5ef371da-31d2-436e-a31b-27408bc91870","Type":"ContainerDied","Data":"9641c012ccf0544e4ed75a925492c5e0798e7cec75383e0ddf3be3efbcd76f1f"}
Nov 02 15:22:19.526345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.526322    5041 generic.go:356] "Generic (PLEG): container finished" podID="c5747cbe-3566-4b3a-83cd-8a2400e832d4" containerID="837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603" exitCode=0
Nov 02 15:22:19.526479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.526391    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s" event={"ID":"c5747cbe-3566-4b3a-83cd-8a2400e832d4","Type":"ContainerDied","Data":"837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603"}
Nov 02 15:22:19.526479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.526439    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s" event={"ID":"c5747cbe-3566-4b3a-83cd-8a2400e832d4","Type":"ContainerDied","Data":"2b34c0f2d43f178ea648659c36271fde6e002925c24a27f70e7e81220184040b"}
Nov 02 15:22:19.526479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.526465    5041 scope.go:122] "RemoveContainer" containerID="837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603"
Nov 02 15:22:19.526479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.526398    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"
Nov 02 15:22:19.530757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.530734    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5747cbe-3566-4b3a-83cd-8a2400e832d4-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:19.530757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.530758    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcnt\" (UniqueName: \"kubernetes.io/projected/c5747cbe-3566-4b3a-83cd-8a2400e832d4-kube-api-access-tvcnt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:19.539585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.539501    5041 scope.go:122] "RemoveContainer" containerID="837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603"
Nov 02 15:22:19.539862 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:19.539835    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603\": container with ID starting with 837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603 not found: ID does not exist" containerID="837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603"
Nov 02 15:22:19.539942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.539873    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603"} err="failed to get container status \"837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603\": rpc error: code = NotFound desc = could not find container \"837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603\": container with ID starting with 837a32c2662ca3c9400c77ff90dac0f487ad1ef86e0cde6a7be9881209530603 not found: ID does not exist"
Nov 02 15:22:19.600055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.599693    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"]
Nov 02 15:22:19.600055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.599760    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3d7df16e-7cbb4b5ddc-lsc6s"]
Nov 02 15:22:19.692752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.692728    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:22:19.733938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.733897    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume\") pod \"5ef371da-31d2-436e-a31b-27408bc91870\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") "
Nov 02 15:22:19.735423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.734219    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l\" (UniqueName: \"kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l\") pod \"5ef371da-31d2-436e-a31b-27408bc91870\" (UID: \"5ef371da-31d2-436e-a31b-27408bc91870\") "
Nov 02 15:22:19.735423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.734295    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume" pod "5ef371da-31d2-436e-a31b-27408bc91870" (UID: "5ef371da-31d2-436e-a31b-27408bc91870"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:19.735423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.734973    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef371da-31d2-436e-a31b-27408bc91870-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:19.736702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.736672    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l" pod "5ef371da-31d2-436e-a31b-27408bc91870" (UID: "5ef371da-31d2-436e-a31b-27408bc91870"). InnerVolumeSpecName "kube-api-access-khc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:19.836920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.836891    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khc8l\" (UniqueName: \"kubernetes.io/projected/5ef371da-31d2-436e-a31b-27408bc91870-kube-api-access-khc8l\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:19.891093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:19.891026    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5747cbe-3566-4b3a-83cd-8a2400e832d4" path="/var/lib/kubelet/pods/c5747cbe-3566-4b3a-83cd-8a2400e832d4/volumes"
Nov 02 15:22:20.538146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:20.538118    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"
Nov 02 15:22:20.538654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:20.538143    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq" event={"ID":"5ef371da-31d2-436e-a31b-27408bc91870","Type":"ContainerDied","Data":"6c3aae97a954b77dfdfda7633a7bd4f1592ccf40ba6d8061509106c574073c88"}
Nov 02 15:22:20.538654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:20.538218    5041 scope.go:122] "RemoveContainer" containerID="9641c012ccf0544e4ed75a925492c5e0798e7cec75383e0ddf3be3efbcd76f1f"
Nov 02 15:22:20.619294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:20.619269    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"]
Nov 02 15:22:20.627863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:20.627836    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d7df16e-7bb7c89f87-rwcpq"]
Nov 02 15:22:21.892039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:21.892015    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef371da-31d2-436e-a31b-27408bc91870" path="/var/lib/kubelet/pods/5ef371da-31d2-436e-a31b-27408bc91870/volumes"
Nov 02 15:22:22.208594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.208567    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:22:22.288101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288020    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288354    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288454    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288628    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288698    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288767    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288839    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288845    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.288981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288870    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.288981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.288965    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket\") pod \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\" (UID: \"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2\") "
Nov 02 15:22:22.289710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289123    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:22.289710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289236    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.289710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289424    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.289710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289497    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.289974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289891    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.289974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289916    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.289974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289930    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.289974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289944    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.289974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.289957    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.293333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.293277    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "kube-api-access-x5nhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:22.293525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.293499    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:22.293814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.293788    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.294940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.294920    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token" pod "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" (UID: "38f6ea88-81e7-4b5f-82e9-c08acfb45ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:22.391599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.391565    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.391599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.391598    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nhc\" (UniqueName: \"kubernetes.io/projected/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-kube-api-access-x5nhc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.391830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.391612    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.391830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.391625    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.566297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.566207    5041 generic.go:356] "Generic (PLEG): container finished" podID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerID="b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035" exitCode=0
Nov 02 15:22:22.566506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.566302    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" event={"ID":"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2","Type":"ContainerDied","Data":"b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035"}
Nov 02 15:22:22.566506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.566311    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"
Nov 02 15:22:22.566506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.566424    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz" event={"ID":"38f6ea88-81e7-4b5f-82e9-c08acfb45ff2","Type":"ContainerDied","Data":"c8ad5c8c756669cf54cc61b60179af48457af3d804bbb559e431eb1307ffb0d4"}
Nov 02 15:22:22.566506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.566460    5041 scope.go:122] "RemoveContainer" containerID="b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035"
Nov 02 15:22:22.584399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.584338    5041 scope.go:122] "RemoveContainer" containerID="b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035"
Nov 02 15:22:22.584716 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:22.584694    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035\": container with ID starting with b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035 not found: ID does not exist" containerID="b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035"
Nov 02 15:22:22.584806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.584722    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035"} err="failed to get container status \"b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035\": rpc error: code = NotFound desc = could not find container \"b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035\": container with ID starting with b00140ed8b1eb9c8137e8850e38142b421413afc2079e3fe9b2ce03d3314f035 not found: ID does not exist"
Nov 02 15:22:22.618287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.618261    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"]
Nov 02 15:22:22.627526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:22.627504    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-xzlgz"]
Nov 02 15:22:23.893937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:23.893901    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" path="/var/lib/kubelet/pods/38f6ea88-81e7-4b5f-82e9-c08acfb45ff2/volumes"
Nov 02 15:22:26.070313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:26.070277    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"]
Nov 02 15:22:26.070705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:26.070494    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy" containerID="cri-o://87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8" gracePeriod=30
Nov 02 15:22:26.690945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:26.690909    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"]
Nov 02 15:22:26.691507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:26.691470    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" containerID="cri-o://98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a" gracePeriod=30
Nov 02 15:22:31.362100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.362074    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:22:31.442076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.442270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442103    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.442330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442300    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.442438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442354    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:31.442438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442402    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.442564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442522    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.442671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442638    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:31.442840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442755    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.442992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.442969    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.443158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.443064    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.443158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.443103    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:31.443158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.443110    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy\") pod \"84f96ca2-28a7-4268-9769-cb798ca68c87\" (UID: \"84f96ca2-28a7-4268-9769-cb798ca68c87\") "
Nov 02 15:22:31.443736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.443618    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:31.443881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.443842    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:31.444938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.444902    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.445054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.444943    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.445054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.444959    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.445054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.444973    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f96ca2-28a7-4268-9769-cb798ca68c87-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.445054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.444986    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.445618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.445587    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:31.445856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.445828    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "kube-api-access-q5x7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:31.446804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.446775    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:31.447515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.447491    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token" pod "84f96ca2-28a7-4268-9769-cb798ca68c87" (UID: "84f96ca2-28a7-4268-9769-cb798ca68c87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:31.546212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.546177    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.546212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.546211    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.546212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.546220    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f96ca2-28a7-4268-9769-cb798ca68c87-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.546475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.546232    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5x7l\" (UniqueName: \"kubernetes.io/projected/84f96ca2-28a7-4268-9769-cb798ca68c87-kube-api-access-q5x7l\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:31.654500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.654420    5041 generic.go:356] "Generic (PLEG): container finished" podID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerID="87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8" exitCode=0
Nov 02 15:22:31.654500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.654484    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"
Nov 02 15:22:31.654660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.654505    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" event={"ID":"84f96ca2-28a7-4268-9769-cb798ca68c87","Type":"ContainerDied","Data":"87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8"}
Nov 02 15:22:31.654660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.654585    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv" event={"ID":"84f96ca2-28a7-4268-9769-cb798ca68c87","Type":"ContainerDied","Data":"4d4c7f9044d4b09111f71e831899b66cbd85439bb26db2796fc7738974e3dd04"}
Nov 02 15:22:31.654660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.654609    5041 scope.go:122] "RemoveContainer" containerID="87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8"
Nov 02 15:22:31.666999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.666980    5041 scope.go:122] "RemoveContainer" containerID="87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8"
Nov 02 15:22:31.667262 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:31.667240    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8\": container with ID starting with 87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8 not found: ID does not exist" containerID="87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8"
Nov 02 15:22:31.667329 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.667274    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8"} err="failed to get container status \"87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8\": rpc error: code = NotFound desc = could not find container \"87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8\": container with ID starting with 87d51a3bde6d86caa556a7123850abe12c2043d92fd8da6344c84adec62d61f8 not found: ID does not exist"
Nov 02 15:22:31.675860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.675838    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:22:31.675938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.675886    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:22:31.703694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.703668    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"]
Nov 02 15:22:31.712011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.711990    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-wtnwv"]
Nov 02 15:22:31.905454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.905383    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" path="/var/lib/kubelet/pods/84f96ca2-28a7-4268-9769-cb798ca68c87/volumes"
Nov 02 15:22:31.958703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:31.958676    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:22:32.053161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053131    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053204    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053252    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053406    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053554    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053649    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.053697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053663    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053755    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053843    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.053937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.053905    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq\") pod \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\" (UID: \"481ebbc3-45aa-4a3e-a1ed-19fc20328400\") "
Nov 02 15:22:32.054948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.054921    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.055752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.055724    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.056215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.056193    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:32.056317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.056266    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:32.056476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.056457    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.057239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.057210    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.059215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.059177    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "kube-api-access-spdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.059293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.059236    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.059775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.059754    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token" pod "481ebbc3-45aa-4a3e-a1ed-19fc20328400" (UID: "481ebbc3-45aa-4a3e-a1ed-19fc20328400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156256    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156290    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdwq\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-kube-api-access-spdwq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156308    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156318    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156326    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156334    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156342    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.156707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.156350    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/481ebbc3-45aa-4a3e-a1ed-19fc20328400-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.250546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.250514    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"]
Nov 02 15:22:32.251373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251347    5041 cpu_manager.go:414] "containerMap: removing container" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy"
Nov 02 15:22:32.251442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251377    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy"
Nov 02 15:22:32.251442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251402    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5ef371da-31d2-436e-a31b-27408bc91870" containerName="payload-pre-processing"
Nov 02 15:22:32.251442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251407    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef371da-31d2-436e-a31b-27408bc91870" containerName="payload-pre-processing"
Nov 02 15:22:32.251442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251423    5041 cpu_manager.go:414] "containerMap: removing container" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy"
Nov 02 15:22:32.251442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251428    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy"
Nov 02 15:22:32.251442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251442    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251447    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251467    5041 cpu_manager.go:414] "containerMap: removing container" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251472    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251483    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251488    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251497    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c5747cbe-3566-4b3a-83cd-8a2400e832d4" containerName="payload-processing"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251502    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5747cbe-3566-4b3a-83cd-8a2400e832d4" containerName="payload-processing"
Nov 02 15:22:32.251618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251616    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f6ea88-81e7-4b5f-82e9-c08acfb45ff2" containerName="istio-proxy"
Nov 02 15:22:32.251859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251626    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1fd59e-e864-4bc6-934e-af5327b9f84f" containerName="istio-proxy"
Nov 02 15:22:32.251859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251638    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f96ca2-28a7-4268-9769-cb798ca68c87" containerName="istio-proxy"
Nov 02 15:22:32.251859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251645    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5665b1-cf1b-42f4-8a2a-d8ac829020cd" containerName="istio-proxy"
Nov 02 15:22:32.251859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251652    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5747cbe-3566-4b3a-83cd-8a2400e832d4" containerName="payload-processing"
Nov 02 15:22:32.251859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251660    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerName="istio-proxy"
Nov 02 15:22:32.251859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.251670    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef371da-31d2-436e-a31b-27408bc91870" containerName="payload-pre-processing"
Nov 02 15:22:32.256190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.256171    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.272665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.272640    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a03e84-serving-cert\""
Nov 02 15:22:32.280835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.280813    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"]
Nov 02 15:22:32.359108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.359077    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.359289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.359264    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxt7\" (UniqueName: \"kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.359345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.359329    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.359441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.359424    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.461298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.461268    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.461813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.461415    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxt7\" (UniqueName: \"kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.461813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.461461    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.461813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.461506    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.461952 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:32.461807    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-serving-cert: secret "maas-api-e2e-inf-a-a03e84-serving-cert" not found
Nov 02 15:22:32.461952 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:32.461902    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls podName:1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7 nodeName:}" failed. No retries permitted until 2026-11-02 15:22:32.961879474 +0000 UTC m=+1605.219948921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls") pod "maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" (UID: "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7") : secret "maas-api-e2e-inf-a-a03e84-serving-cert" not found
Nov 02 15:22:32.462353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.462328    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.464189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.464167    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.482867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.482845    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxt7\" (UniqueName: \"kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.667837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.667806    5041 generic.go:356] "Generic (PLEG): container finished" podID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" containerID="98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a" exitCode=0
Nov 02 15:22:32.668031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.667919    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"
Nov 02 15:22:32.668031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.667952    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" event={"ID":"481ebbc3-45aa-4a3e-a1ed-19fc20328400","Type":"ContainerDied","Data":"98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a"}
Nov 02 15:22:32.668031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.668010    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj" event={"ID":"481ebbc3-45aa-4a3e-a1ed-19fc20328400","Type":"ContainerDied","Data":"c96795415b87280a9cd0b3924ffacd4f9ececee215f2d107dab5f65a25b0afe0"}
Nov 02 15:22:32.668257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.668039    5041 scope.go:122] "RemoveContainer" containerID="98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a"
Nov 02 15:22:32.683001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.682978    5041 scope.go:122] "RemoveContainer" containerID="98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a"
Nov 02 15:22:32.683274 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:32.683250    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a\": container with ID starting with 98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a not found: ID does not exist" containerID="98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a"
Nov 02 15:22:32.683351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.683283    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a"} err="failed to get container status \"98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a\": rpc error: code = NotFound desc = could not find container \"98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a\": container with ID starting with 98e14b9c93185588459894e4d8c5ab6d20612607edef6a6a378a522ae711329a not found: ID does not exist"
Nov 02 15:22:32.725491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.725426    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"]
Nov 02 15:22:32.728304 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:32.728283    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ebbc3_45aa_4a3e_a1ed_19fc20328400.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 15:22:32.732168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.732150    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-bn8pj"]
Nov 02 15:22:32.987199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.987133    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:32.989483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:32.989462    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:33.173565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:33.173534    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:33.633935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:33.633902    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"]
Nov 02 15:22:33.637466 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:22:33.637431    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3b9e9a_3d9c_4662_8026_6a93d91fbcd7.slice/crio-7c869b7b6bed799ee8728d3ff5805b13eca6f14fbea3727dad73a9ff0d929eb6 WatchSource:0}: Error finding container 7c869b7b6bed799ee8728d3ff5805b13eca6f14fbea3727dad73a9ff0d929eb6: Status 404 returned error can't find the container with id 7c869b7b6bed799ee8728d3ff5805b13eca6f14fbea3727dad73a9ff0d929eb6
Nov 02 15:22:33.687166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:33.687102    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" event={"ID":"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7","Type":"ContainerStarted","Data":"7c869b7b6bed799ee8728d3ff5805b13eca6f14fbea3727dad73a9ff0d929eb6"}
Nov 02 15:22:33.917627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:33.917528    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481ebbc3-45aa-4a3e-a1ed-19fc20328400" path="/var/lib/kubelet/pods/481ebbc3-45aa-4a3e-a1ed-19fc20328400/volumes"
Nov 02 15:22:34.699108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:34.699050    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" event={"ID":"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7","Type":"ContainerStarted","Data":"f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d"}
Nov 02 15:22:34.699610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:34.699155    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:34.740661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:34.740614    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" podStartSLOduration=2.408681153 podStartE2EDuration="2.740597763s" totalImagesPullingTime="331.91661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:34.735563201 +0000 UTC m=+1606.993632652" watchObservedRunningTime="2026-11-02 15:22:34.740597763 +0000 UTC m=+1606.998667214"
Nov 02 15:22:36.775610 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:22:36.775566    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:22:37.725444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.725409    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"]
Nov 02 15:22:37.730648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.730628    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.756716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.756688    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"]
Nov 02 15:22:37.797918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.797886    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.798600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.797991    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.798600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.798172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.798600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.798270    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bsw6\" (UniqueName: \"kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.808234 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.808207    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4ec02d-serving-cert\""
Nov 02 15:22:37.900778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.900747    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bsw6\" (UniqueName: \"kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.900950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.900798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.900950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.900876    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.901085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.901064    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.901754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.901728    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.903257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.903237    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.903626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.903609    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:37.921790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:37.921766    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bsw6\" (UniqueName: \"kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6\") pod \"maas-api-e2e-api-b-4ec02d-845df4848-ndgvc\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:38.063914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:38.063598    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:38.397256 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:22:38.397221    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267d929a_319e_4acc_8494_83f1b42cedd1.slice/crio-e85015d591e7a0909cc5d6e4b4ca11ec178d98d4969e4fd9d406c945398b3567 WatchSource:0}: Error finding container e85015d591e7a0909cc5d6e4b4ca11ec178d98d4969e4fd9d406c945398b3567: Status 404 returned error can't find the container with id e85015d591e7a0909cc5d6e4b4ca11ec178d98d4969e4fd9d406c945398b3567
Nov 02 15:22:38.399625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:38.399603    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"]
Nov 02 15:22:38.738389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:38.738324    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc" event={"ID":"267d929a-319e-4acc-8494-83f1b42cedd1","Type":"ContainerStarted","Data":"e85015d591e7a0909cc5d6e4b4ca11ec178d98d4969e4fd9d406c945398b3567"}
Nov 02 15:22:39.761154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:39.761024    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc" event={"ID":"267d929a-319e-4acc-8494-83f1b42cedd1","Type":"ContainerStarted","Data":"c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75"}
Nov 02 15:22:39.761869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:39.761850    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:39.946131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:39.925950    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc" podStartSLOduration=2.562034188 podStartE2EDuration="2.925928019s" totalImagesPullingTime="363.893831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:39.80977304 +0000 UTC m=+1612.067842492" watchObservedRunningTime="2026-11-02 15:22:39.925928019 +0000 UTC m=+1612.183997470"
Nov 02 15:22:40.713617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:40.713589    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:22:45.777920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:45.777887    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:22:56.206044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:56.206005    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"]
Nov 02 15:22:56.206883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:22:56.206321    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy" containerID="cri-o://153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0" gracePeriod=30
Nov 02 15:23:01.515391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.515348    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:23:01.607447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607408    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.607616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607475    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.607699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607668    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.607770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607702    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.607770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607742    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.607874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607799    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.607874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607858    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.607987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.607910    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.608100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.608079    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.608309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.608289    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy\") pod \"8ba23440-bf0b-40e7-9314-63c85090cacd\" (UID: \"8ba23440-bf0b-40e7-9314-63c85090cacd\") "
Nov 02 15:23:01.608588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.608135    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.608588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.608166    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:01.608588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.608219    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.608588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.608300    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.609228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.609205    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.609330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.609234    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.609330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.609250    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba23440-bf0b-40e7-9314-63c85090cacd-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.609330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.609262    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.609330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.609275    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.610642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.610616    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:01.611302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.611029    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "kube-api-access-zxqkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:01.611745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.611720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.612332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.612306    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token" pod "8ba23440-bf0b-40e7-9314-63c85090cacd" (UID: "8ba23440-bf0b-40e7-9314-63c85090cacd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:01.710576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.710545    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.710576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.710575    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqkx\" (UniqueName: \"kubernetes.io/projected/8ba23440-bf0b-40e7-9314-63c85090cacd-kube-api-access-zxqkx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.710785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.710586    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.710785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.710596    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba23440-bf0b-40e7-9314-63c85090cacd-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.792738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.792672    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"]
Nov 02 15:23:01.793569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.793554    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy"
Nov 02 15:23:01.793615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.793570    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy"
Nov 02 15:23:01.793744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.793735    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerName="istio-proxy"
Nov 02 15:23:01.797940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.797922    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:01.812917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.812894    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2abeaa\""
Nov 02 15:23:01.813671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.813648    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2abeaa-dockercfg-v9s58\""
Nov 02 15:23:01.823751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.823724    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"]
Nov 02 15:23:01.997387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:01.971994    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mjk\" (UniqueName: \"kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk\") pod \"payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.045553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.040232    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.213727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.213681    5041 generic.go:356] "Generic (PLEG): container finished" podID="8ba23440-bf0b-40e7-9314-63c85090cacd" containerID="153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0" exitCode=0
Nov 02 15:23:02.214199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.214172    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"
Nov 02 15:23:02.291144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.288770    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mjk\" (UniqueName: \"kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk\") pod \"payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.291144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.288835    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.304953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.304870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.318802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.318771    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"]
Nov 02 15:23:02.326238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.326210    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" event={"ID":"8ba23440-bf0b-40e7-9314-63c85090cacd","Type":"ContainerDied","Data":"153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0"}
Nov 02 15:23:02.326384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.326241    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg" event={"ID":"8ba23440-bf0b-40e7-9314-63c85090cacd","Type":"ContainerDied","Data":"e932f940aeb7ba60ecde3c80c0834ca858a795df4df678052917d4d04218dd9d"}
Nov 02 15:23:02.326384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.326254    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"]
Nov 02 15:23:02.326384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.326341    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.326384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.326345    5041 scope.go:122] "RemoveContainer" containerID="153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0"
Nov 02 15:23:02.332060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.332039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mjk\" (UniqueName: \"kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk\") pod \"payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.351873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.351809    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"]
Nov 02 15:23:02.357458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.357437    5041 scope.go:122] "RemoveContainer" containerID="153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0"
Nov 02 15:23:02.357743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.357725    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-nfpfg"]
Nov 02 15:23:02.357827 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:23:02.357765    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0\": container with ID starting with 153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0 not found: ID does not exist" containerID="153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0"
Nov 02 15:23:02.357827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.357800    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0"} err="failed to get container status \"153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0\": rpc error: code = NotFound desc = could not find container \"153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0\": container with ID starting with 153a47976523d2c1c99b513ab1824e6a19b8b02f82f416ed01864026fb6cdac0 not found: ID does not exist"
Nov 02 15:23:02.390574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.390547    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm247\" (UniqueName: \"kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247\") pod \"payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.390692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.390675    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.415230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.415210    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:02.494335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.493200    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm247\" (UniqueName: \"kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247\") pod \"payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.494335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.493312    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.494545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.494487    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.515254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.515226    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm247\" (UniqueName: \"kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247\") pod \"payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.664250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.664172    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:02.724790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.724760    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"]
Nov 02 15:23:02.725971 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:23:02.725945    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836a1569_607e_49e7_8af2_b9053eda759c.slice/crio-4aeb4dda64974a0923753b79368902acf949c0a9ccff2d5bc9a79013dee66285 WatchSource:0}: Error finding container 4aeb4dda64974a0923753b79368902acf949c0a9ccff2d5bc9a79013dee66285: Status 404 returned error can't find the container with id 4aeb4dda64974a0923753b79368902acf949c0a9ccff2d5bc9a79013dee66285
Nov 02 15:23:02.986245 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:23:02.986219    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb38061_bac4_4186_9588_0e297d26fb1e.slice/crio-dfa2b02e26e124a70de5a25f7a68a877120eaeee1c0dd95dc6ba94b4b379663e WatchSource:0}: Error finding container dfa2b02e26e124a70de5a25f7a68a877120eaeee1c0dd95dc6ba94b4b379663e: Status 404 returned error can't find the container with id dfa2b02e26e124a70de5a25f7a68a877120eaeee1c0dd95dc6ba94b4b379663e
Nov 02 15:23:02.988119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:02.988095    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"]
Nov 02 15:23:03.226192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.226157    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" event={"ID":"836a1569-607e-49e7-8af2-b9053eda759c","Type":"ContainerStarted","Data":"27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536"}
Nov 02 15:23:03.226395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.226219    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" event={"ID":"836a1569-607e-49e7-8af2-b9053eda759c","Type":"ContainerStarted","Data":"4aeb4dda64974a0923753b79368902acf949c0a9ccff2d5bc9a79013dee66285"}
Nov 02 15:23:03.226479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.226402    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:03.228077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.228055    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk" event={"ID":"cdb38061-bac4-4186-9588-0e297d26fb1e","Type":"ContainerStarted","Data":"97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b"}
Nov 02 15:23:03.228200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.228088    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk" event={"ID":"cdb38061-bac4-4186-9588-0e297d26fb1e","Type":"ContainerStarted","Data":"dfa2b02e26e124a70de5a25f7a68a877120eaeee1c0dd95dc6ba94b4b379663e"}
Nov 02 15:23:03.228200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.228182    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:03.267564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.267484    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" podStartSLOduration=2.267472029 podStartE2EDuration="2.267472029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:03.26490585 +0000 UTC m=+1635.522975300" watchObservedRunningTime="2026-11-02 15:23:03.267472029 +0000 UTC m=+1635.525541525"
Nov 02 15:23:03.302641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.302602    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk" podStartSLOduration=2.302592351 podStartE2EDuration="2.302592351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:03.300944244 +0000 UTC m=+1635.559013694" watchObservedRunningTime="2026-11-02 15:23:03.302592351 +0000 UTC m=+1635.560661800"
Nov 02 15:23:03.893438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:03.893403    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba23440-bf0b-40e7-9314-63c85090cacd" path="/var/lib/kubelet/pods/8ba23440-bf0b-40e7-9314-63c85090cacd/volumes"
Nov 02 15:23:06.772073 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:23:06.772032    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:23:14.262569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:14.262536    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:23:14.264489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:14.264464    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:23:31.404864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:31.404782    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"]
Nov 02 15:23:31.405445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:31.404986    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy" containerID="cri-o://bd3b7c106d4c36c53a87362d39fdb4fd0b102ee8ef784bad6dce83a5e91e42e4" gracePeriod=30
Nov 02 15:23:31.428989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:31.428954    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"]
Nov 02 15:23:31.429177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:31.429154    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy" containerID="cri-o://e5545b813ad2f22bc08a4ed3a030a4a3058ed937badc3c9e82cde24c20ba6278" gracePeriod=30
Nov 02 15:23:31.440538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:31.440508    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"]
Nov 02 15:23:31.440707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:31.440689    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy" containerID="cri-o://145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa" gracePeriod=30
Nov 02 15:23:32.716846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.716812    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"]
Nov 02 15:23:32.722990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.722968    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:32.747405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.744889    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3daba5\""
Nov 02 15:23:32.747405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.745014    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3daba5-dockercfg-nlzqp\""
Nov 02 15:23:32.753405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.753319    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"]
Nov 02 15:23:32.930385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.917146    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtc9\" (UniqueName: \"kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9\") pod \"payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:32.930385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.917260    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:32.930385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.928421    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"]
Nov 02 15:23:32.957390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.956845    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.007391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:32.993136    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"]
Nov 02 15:23:33.026389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.024444    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtc9\" (UniqueName: \"kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9\") pod \"payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:33.026389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.024585    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:33.045388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.029195    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:33.110846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.110809    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtc9\" (UniqueName: \"kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9\") pod \"payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:33.138424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.137013    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdv88\" (UniqueName: \"kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88\") pod \"payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.138424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.137260    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.239212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.239129    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdv88\" (UniqueName: \"kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88\") pod \"payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.239421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.239267    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.239979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.239956    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.263638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.263607    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdv88\" (UniqueName: \"kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88\") pod \"payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.323831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.323808    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:33.341241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.339762    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:33.653337 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:23:33.653308    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf019e4_176c_4e5e_bd8d_a8c7bfd55171.slice/crio-740ea4507d6dd7e38e5a6c8fed069eb5604e242120fae2d7f230c565a908de2c WatchSource:0}: Error finding container 740ea4507d6dd7e38e5a6c8fed069eb5604e242120fae2d7f230c565a908de2c: Status 404 returned error can't find the container with id 740ea4507d6dd7e38e5a6c8fed069eb5604e242120fae2d7f230c565a908de2c
Nov 02 15:23:33.654729 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.654706    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"]
Nov 02 15:23:33.673501 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:23:33.673471    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9410559_197b_400c_9680_a61e0127df19.slice/crio-cbdcb7cfd7890a1ad08baf4aa2e01ac42b8938e4abaccbba1bdb539f157bdb18 WatchSource:0}: Error finding container cbdcb7cfd7890a1ad08baf4aa2e01ac42b8938e4abaccbba1bdb539f157bdb18: Status 404 returned error can't find the container with id cbdcb7cfd7890a1ad08baf4aa2e01ac42b8938e4abaccbba1bdb539f157bdb18
Nov 02 15:23:33.674355 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.674329    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"]
Nov 02 15:23:33.725096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.725065    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d" event={"ID":"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171","Type":"ContainerStarted","Data":"740ea4507d6dd7e38e5a6c8fed069eb5604e242120fae2d7f230c565a908de2c"}
Nov 02 15:23:33.726612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:33.726588    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" event={"ID":"b9410559-197b-400c-9680-a61e0127df19","Type":"ContainerStarted","Data":"cbdcb7cfd7890a1ad08baf4aa2e01ac42b8938e4abaccbba1bdb539f157bdb18"}
Nov 02 15:23:34.737028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:34.736985    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d" event={"ID":"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171","Type":"ContainerStarted","Data":"8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb"}
Nov 02 15:23:34.737560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:34.737216    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:34.738754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:34.738726    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" event={"ID":"b9410559-197b-400c-9680-a61e0127df19","Type":"ContainerStarted","Data":"c9d7d3ddf3976cd2efb9f73d00468d6a7ed8b9904579bb1d6e3cc8e5d5306bc6"}
Nov 02 15:23:34.738862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:34.738805    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:34.774597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:34.774553    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d" podStartSLOduration=2.774538063 podStartE2EDuration="2.774538063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:34.771697377 +0000 UTC m=+1667.029766827" watchObservedRunningTime="2026-11-02 15:23:34.774538063 +0000 UTC m=+1667.032607515"
Nov 02 15:23:34.807394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:34.807333    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" podStartSLOduration=2.807318568 podStartE2EDuration="2.807318568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:34.806263134 +0000 UTC m=+1667.064332583" watchObservedRunningTime="2026-11-02 15:23:34.807318568 +0000 UTC m=+1667.065388022"
Nov 02 15:23:36.774457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.773354    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:23:36.786428 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:23:36.786391    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:23:36.791493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.791464    5041 generic.go:356] "Generic (PLEG): container finished" podID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerID="e5545b813ad2f22bc08a4ed3a030a4a3058ed937badc3c9e82cde24c20ba6278" exitCode=0
Nov 02 15:23:36.791642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.791537    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" event={"ID":"5d260c65-7fb3-4dbc-96b8-e1a60c44e655","Type":"ContainerDied","Data":"e5545b813ad2f22bc08a4ed3a030a4a3058ed937badc3c9e82cde24c20ba6278"}
Nov 02 15:23:36.796090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.794581    5041 generic.go:356] "Generic (PLEG): container finished" podID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerID="bd3b7c106d4c36c53a87362d39fdb4fd0b102ee8ef784bad6dce83a5e91e42e4" exitCode=0
Nov 02 15:23:36.796090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.794851    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" event={"ID":"d062b215-806e-4bef-a3cf-a5f42a1492e4","Type":"ContainerDied","Data":"bd3b7c106d4c36c53a87362d39fdb4fd0b102ee8ef784bad6dce83a5e91e42e4"}
Nov 02 15:23:36.798644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.798621    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:23:36.800089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.800067    5041 generic.go:356] "Generic (PLEG): container finished" podID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerID="145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa" exitCode=0
Nov 02 15:23:36.800219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.800107    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" event={"ID":"fa384528-cbbf-43e7-85ec-ca9c409d2b4e","Type":"ContainerDied","Data":"145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa"}
Nov 02 15:23:36.800219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.800110    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"
Nov 02 15:23:36.800219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.800139    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb" event={"ID":"fa384528-cbbf-43e7-85ec-ca9c409d2b4e","Type":"ContainerDied","Data":"a724e0f4fcfb509f375b609192081d3f6955fc36be9e176e304e034c95d94abb"}
Nov 02 15:23:36.800219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.800160    5041 scope.go:122] "RemoveContainer" containerID="145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa"
Nov 02 15:23:36.819203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.819182    5041 scope.go:122] "RemoveContainer" containerID="145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa"
Nov 02 15:23:36.819650 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:23:36.819563    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa\": container with ID starting with 145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa not found: ID does not exist" containerID="145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa"
Nov 02 15:23:36.819650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.819601    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa"} err="failed to get container status \"145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa\": rpc error: code = NotFound desc = could not find container \"145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa\": container with ID starting with 145b8428b1012ad3e1311e5f32848cf803672884b7b3bc0b1d78848c5dbeb6aa not found: ID does not exist"
Nov 02 15:23:36.901815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.901785    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:23:36.904095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904063    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.904253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904107    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.904253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904143    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.904397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904379    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.904480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904462    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.904540 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904521    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.904599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.904587    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.905030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905000    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.905133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905004    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.905193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905139    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.905285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905207    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.905285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905254    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.905428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905326    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.905484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905432    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.905681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905656    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.905681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905670    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.905820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905701    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.905820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905768    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.905936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905855    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs\") pod \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\" (UID: \"5d260c65-7fb3-4dbc-96b8-e1a60c44e655\") "
Nov 02 15:23:36.905936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905916    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.906050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.905957    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket\") pod \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\" (UID: \"fa384528-cbbf-43e7-85ec-ca9c409d2b4e\") "
Nov 02 15:23:36.906658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.906623    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.906658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.906647    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.906658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.906657    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.907351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.906812    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.909269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909239    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.909418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909312    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "kube-api-access-cc4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.909536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909513    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.909594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909534    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.909723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909705    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.909723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909714    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.909903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.909884    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.910721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.910696    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.911271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.911239    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.911927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.911900    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "kube-api-access-2qjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.912331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.912294    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.913885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.913836    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.914178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.914152    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo" pod "5d260c65-7fb3-4dbc-96b8-e1a60c44e655" (UID: "5d260c65-7fb3-4dbc-96b8-e1a60c44e655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.914421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:36.914402    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy" pod "fa384528-cbbf-43e7-85ec-ca9c409d2b4e" (UID: "fa384528-cbbf-43e7-85ec-ca9c409d2b4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.008075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008000    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008109    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008163    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008202    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008258    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008268    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.008418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008306    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008382    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008419    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.008659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008433    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008548    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo\") pod \"d062b215-806e-4bef-a3cf-a5f42a1492e4\" (UID: \"d062b215-806e-4bef-a3cf-a5f42a1492e4\") "
Nov 02 15:23:37.008873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008658    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.008873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008665    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:37.008963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.008868    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.009484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009438    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjvg\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-kube-api-access-2qjvg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009469    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009486    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009500    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009513    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009526    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d062b215-806e-4bef-a3cf-a5f42a1492e4-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009539    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009551    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009564    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009578    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc4xt\" (UniqueName: \"kubernetes.io/projected/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-kube-api-access-cc4xt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009592    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009608    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009621    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009635    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009648    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009662    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009675    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa384528-cbbf-43e7-85ec-ca9c409d2b4e-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009691    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009705    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.009874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.009718    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d260c65-7fb3-4dbc-96b8-e1a60c44e655-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.010591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.010570    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:37.010669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.010647    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "kube-api-access-r8cbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:37.010852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.010826    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.010852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.010831    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token" pod "d062b215-806e-4bef-a3cf-a5f42a1492e4" (UID: "d062b215-806e-4bef-a3cf-a5f42a1492e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:37.110832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.110801    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.110832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.110830    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8cbn\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-kube-api-access-r8cbn\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.111074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.110840    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.111074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.110849    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d062b215-806e-4bef-a3cf-a5f42a1492e4-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.145058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.145029    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"]
Nov 02 15:23:37.156489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.156456    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-szsrb"]
Nov 02 15:23:37.810910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.810805    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"
Nov 02 15:23:37.810910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.810815    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx" event={"ID":"d062b215-806e-4bef-a3cf-a5f42a1492e4","Type":"ContainerDied","Data":"2d88cf558a3f486f330940578b14f5ede73a0710d2f348df9dd49ff28344c36a"}
Nov 02 15:23:37.810910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.810879    5041 scope.go:122] "RemoveContainer" containerID="bd3b7c106d4c36c53a87362d39fdb4fd0b102ee8ef784bad6dce83a5e91e42e4"
Nov 02 15:23:37.814889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.814813    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx" event={"ID":"5d260c65-7fb3-4dbc-96b8-e1a60c44e655","Type":"ContainerDied","Data":"93a1d2b57e728771870597fc60a6ebb67081ec1cb456ce9eec0c2d081ce1db4f"}
Nov 02 15:23:37.814889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.814852    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"
Nov 02 15:23:37.824690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.824667    5041 scope.go:122] "RemoveContainer" containerID="e5545b813ad2f22bc08a4ed3a030a4a3058ed937badc3c9e82cde24c20ba6278"
Nov 02 15:23:37.871777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.871755    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"]
Nov 02 15:23:37.890708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.890681    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" path="/var/lib/kubelet/pods/fa384528-cbbf-43e7-85ec-ca9c409d2b4e/volumes"
Nov 02 15:23:37.891065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.891051    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-chbbx"]
Nov 02 15:23:37.975482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:37.975413    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"]
Nov 02 15:23:38.069993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:38.069865    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-sdwjx"]
Nov 02 15:23:39.890443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:39.890406    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" path="/var/lib/kubelet/pods/5d260c65-7fb3-4dbc-96b8-e1a60c44e655/volumes"
Nov 02 15:23:39.891026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:39.891009    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" path="/var/lib/kubelet/pods/d062b215-806e-4bef-a3cf-a5f42a1492e4/volumes"
Nov 02 15:23:45.751823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:45.751793    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:23:45.752225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:45.752197    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:23:57.613221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:23:57.613185    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"]
Nov 02 15:24:02.614405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:02.614339    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm" podUID="4783afee-ed79-4776-ab97-289b6f26c7ae" containerName="payload-processing" containerID="cri-o://1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f" gracePeriod=25
Nov 02 15:24:02.953033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:02.953006    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:24:03.020608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.020573    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume\") pod \"4783afee-ed79-4776-ab97-289b6f26c7ae\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") "
Nov 02 15:24:03.020889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.020871    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf\" (UniqueName: \"kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf\") pod \"4783afee-ed79-4776-ab97-289b6f26c7ae\" (UID: \"4783afee-ed79-4776-ab97-289b6f26c7ae\") "
Nov 02 15:24:03.021044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.021015    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume" pod "4783afee-ed79-4776-ab97-289b6f26c7ae" (UID: "4783afee-ed79-4776-ab97-289b6f26c7ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:03.021451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.021430    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4783afee-ed79-4776-ab97-289b6f26c7ae-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:03.022925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.022906    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf" pod "4783afee-ed79-4776-ab97-289b6f26c7ae" (UID: "4783afee-ed79-4776-ab97-289b6f26c7ae"). InnerVolumeSpecName "kube-api-access-jxwqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:03.121702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.121667    5041 generic.go:356] "Generic (PLEG): container finished" podID="4783afee-ed79-4776-ab97-289b6f26c7ae" containerID="1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f" exitCode=0
Nov 02 15:24:03.121858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.121705    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm" event={"ID":"4783afee-ed79-4776-ab97-289b6f26c7ae","Type":"ContainerDied","Data":"1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f"}
Nov 02 15:24:03.121858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.121725    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"
Nov 02 15:24:03.121858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.121805    5041 scope.go:122] "RemoveContainer" containerID="1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f"
Nov 02 15:24:03.122030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.121795    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm" event={"ID":"4783afee-ed79-4776-ab97-289b6f26c7ae","Type":"ContainerDied","Data":"e67a7658d21671d560e2cfa62cb7185965e1ffd80a44821d26edce4d7f4f6a2d"}
Nov 02 15:24:03.122912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.122828    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxwqf\" (UniqueName: \"kubernetes.io/projected/4783afee-ed79-4776-ab97-289b6f26c7ae-kube-api-access-jxwqf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:03.134027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.134007    5041 scope.go:122] "RemoveContainer" containerID="1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f"
Nov 02 15:24:03.134321 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:03.134301    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f\": container with ID starting with 1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f not found: ID does not exist" containerID="1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f"
Nov 02 15:24:03.134407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.134328    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f"} err="failed to get container status \"1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f\": rpc error: code = NotFound desc = could not find container \"1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f\": container with ID starting with 1e7824cb985f68784fa1c81840af599c0d1ce38fb29854eb909bc79adda8313f not found: ID does not exist"
Nov 02 15:24:03.169322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.169298    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"]
Nov 02 15:24:03.174143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.174120    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-815746-78756d469f-lcxkm"]
Nov 02 15:24:03.941388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:03.930279    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4783afee-ed79-4776-ab97-289b6f26c7ae" path="/var/lib/kubelet/pods/4783afee-ed79-4776-ab97-289b6f26c7ae/volumes"
Nov 02 15:24:06.769840 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:06.769802    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:24:07.421508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:07.421473    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"]
Nov 02 15:24:07.464846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:07.464812    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"]
Nov 02 15:24:12.426473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:12.426424    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm" podUID="23103073-28d9-454e-b080-3ce8a4caaae9" containerName="payload-processing" containerID="cri-o://4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59" gracePeriod=25
Nov 02 15:24:12.473387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:12.467136    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg" podUID="96ceb140-f94a-46f0-a34d-54b9e542ccdb" containerName="payload-pre-processing" containerID="cri-o://09abd27254c60c98890628cf6737bb5a12069c3f0ec03a5b3e6f657e1badbb41" gracePeriod=25
Nov 02 15:24:13.304541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.303698    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:24:13.313382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.312807    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl\" (UniqueName: \"kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl\") pod \"23103073-28d9-454e-b080-3ce8a4caaae9\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") "
Nov 02 15:24:13.313382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.313143    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume\") pod \"23103073-28d9-454e-b080-3ce8a4caaae9\" (UID: \"23103073-28d9-454e-b080-3ce8a4caaae9\") "
Nov 02 15:24:13.341493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.333073    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume" pod "23103073-28d9-454e-b080-3ce8a4caaae9" (UID: "23103073-28d9-454e-b080-3ce8a4caaae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:13.406419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.403202    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl" pod "23103073-28d9-454e-b080-3ce8a4caaae9" (UID: "23103073-28d9-454e-b080-3ce8a4caaae9"). InnerVolumeSpecName "kube-api-access-sppvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:13.435179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.428197    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23103073-28d9-454e-b080-3ce8a4caaae9-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:13.435179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.428231    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppvl\" (UniqueName: \"kubernetes.io/projected/23103073-28d9-454e-b080-3ce8a4caaae9-kube-api-access-sppvl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:13.506967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.504705    5041 generic.go:356] "Generic (PLEG): container finished" podID="23103073-28d9-454e-b080-3ce8a4caaae9" containerID="4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59" exitCode=0
Nov 02 15:24:13.506967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.504765    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"
Nov 02 15:24:13.506967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.504859    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm" event={"ID":"23103073-28d9-454e-b080-3ce8a4caaae9","Type":"ContainerDied","Data":"4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59"}
Nov 02 15:24:13.506967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.504911    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm" event={"ID":"23103073-28d9-454e-b080-3ce8a4caaae9","Type":"ContainerDied","Data":"31bea0217ccdc55243b37db3937636c850b97f569c6e9f7dc26e92de8aa35821"}
Nov 02 15:24:13.506967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.504945    5041 scope.go:122] "RemoveContainer" containerID="4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59"
Nov 02 15:24:13.508669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.508636    5041 generic.go:356] "Generic (PLEG): container finished" podID="96ceb140-f94a-46f0-a34d-54b9e542ccdb" containerID="09abd27254c60c98890628cf6737bb5a12069c3f0ec03a5b3e6f657e1badbb41" exitCode=0
Nov 02 15:24:13.508832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.508813    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg" event={"ID":"96ceb140-f94a-46f0-a34d-54b9e542ccdb","Type":"ContainerDied","Data":"09abd27254c60c98890628cf6737bb5a12069c3f0ec03a5b3e6f657e1badbb41"}
Nov 02 15:24:13.610587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.610422    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"]
Nov 02 15:24:13.610671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.610652    5041 scope.go:122] "RemoveContainer" containerID="4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59"
Nov 02 15:24:13.611133 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:13.611107    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59\": container with ID starting with 4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59 not found: ID does not exist" containerID="4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59"
Nov 02 15:24:13.611290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.611266    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59"} err="failed to get container status \"4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59\": rpc error: code = NotFound desc = could not find container \"4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59\": container with ID starting with 4cdf17f613e096283817c3b5214580bb490e15bb5dd775f0cdc7a4ee68fdde59 not found: ID does not exist"
Nov 02 15:24:13.688389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.667938    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db75ff-84cddcdf95-hp8nm"]
Nov 02 15:24:13.702690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.702664    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:24:13.879442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.879103    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume\") pod \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") "
Nov 02 15:24:13.879442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.879276    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9\" (UniqueName: \"kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9\") pod \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\" (UID: \"96ceb140-f94a-46f0-a34d-54b9e542ccdb\") "
Nov 02 15:24:13.890418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.887546    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9" pod "96ceb140-f94a-46f0-a34d-54b9e542ccdb" (UID: "96ceb140-f94a-46f0-a34d-54b9e542ccdb"). InnerVolumeSpecName "kube-api-access-zzxz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:13.890418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.887927    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume" pod "96ceb140-f94a-46f0-a34d-54b9e542ccdb" (UID: "96ceb140-f94a-46f0-a34d-54b9e542ccdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:13.964407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.962174    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23103073-28d9-454e-b080-3ce8a4caaae9" path="/var/lib/kubelet/pods/23103073-28d9-454e-b080-3ce8a4caaae9/volumes"
Nov 02 15:24:13.988674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.988057    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ceb140-f94a-46f0-a34d-54b9e542ccdb-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:13.988674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:13.988090    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxz9\" (UniqueName: \"kubernetes.io/projected/96ceb140-f94a-46f0-a34d-54b9e542ccdb-kube-api-access-zzxz9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:14.073613 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:14.073119    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ceb140_f94a_46f0_a34d_54b9e542ccdb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ceb140_f94a_46f0_a34d_54b9e542ccdb.slice/crio-68424e918a0d99fa7e11a941e76e26fd9e4844b64db0a7e3a7c3d382fd68a0a4\": RecentStats: unable to find data in memory cache]"
Nov 02 15:24:14.519890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:14.519859    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"
Nov 02 15:24:14.520404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:14.519891    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg" event={"ID":"96ceb140-f94a-46f0-a34d-54b9e542ccdb","Type":"ContainerDied","Data":"68424e918a0d99fa7e11a941e76e26fd9e4844b64db0a7e3a7c3d382fd68a0a4"}
Nov 02 15:24:14.520404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:14.519960    5041 scope.go:122] "RemoveContainer" containerID="09abd27254c60c98890628cf6737bb5a12069c3f0ec03a5b3e6f657e1badbb41"
Nov 02 15:24:14.582161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:14.582134    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"]
Nov 02 15:24:14.595420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:14.595383    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db75ff-ffb9c85dc-vtjvg"]
Nov 02 15:24:15.890648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:15.890617    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ceb140-f94a-46f0-a34d-54b9e542ccdb" path="/var/lib/kubelet/pods/96ceb140-f94a-46f0-a34d-54b9e542ccdb/volumes"
Nov 02 15:24:16.552920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.552885    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"]
Nov 02 15:24:16.553787 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553770    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy"
Nov 02 15:24:16.553866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553789    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy"
Nov 02 15:24:16.553866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553805    5041 cpu_manager.go:414] "containerMap: removing container" podUID="23103073-28d9-454e-b080-3ce8a4caaae9" containerName="payload-processing"
Nov 02 15:24:16.553866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553811    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23103073-28d9-454e-b080-3ce8a4caaae9" containerName="payload-processing"
Nov 02 15:24:16.553866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553824    5041 cpu_manager.go:414] "containerMap: removing container" podUID="96ceb140-f94a-46f0-a34d-54b9e542ccdb" containerName="payload-pre-processing"
Nov 02 15:24:16.553866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553829    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ceb140-f94a-46f0-a34d-54b9e542ccdb" containerName="payload-pre-processing"
Nov 02 15:24:16.553866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553866    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553871    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553883    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4783afee-ed79-4776-ab97-289b6f26c7ae" containerName="payload-processing"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553888    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4783afee-ed79-4776-ab97-289b6f26c7ae" containerName="payload-processing"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553897    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.553902    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.554018    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d260c65-7fb3-4dbc-96b8-e1a60c44e655" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.554026    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4783afee-ed79-4776-ab97-289b6f26c7ae" containerName="payload-processing"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.554034    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa384528-cbbf-43e7-85ec-ca9c409d2b4e" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.554043    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d062b215-806e-4bef-a3cf-a5f42a1492e4" containerName="istio-proxy"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.554050    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="23103073-28d9-454e-b080-3ce8a4caaae9" containerName="payload-processing"
Nov 02 15:24:16.554058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.554057    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ceb140-f94a-46f0-a34d-54b9e542ccdb" containerName="payload-pre-processing"
Nov 02 15:24:16.559829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.559809    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.582571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.582542    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3daba5-serving-cert\""
Nov 02 15:24:16.590576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.590547    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"]
Nov 02 15:24:16.630649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.630611    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.630816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.630739    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmzc\" (UniqueName: \"kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.630876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.630841    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.630916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.630881    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.732927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.732884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.733157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.733141    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.733238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.733224    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmzc\" (UniqueName: \"kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.733322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.733307    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.733635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.733611    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.735823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.735801    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.735916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.735875    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.754135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.754111    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmzc\" (UniqueName: \"kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc\") pod \"maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:16.882349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:16.880995    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:17.222680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:17.222646    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"]
Nov 02 15:24:17.223050 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:24:17.222757    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7831b98_9674_4352_8847_f1119617ae99.slice/crio-555c51faa372df7e73d4c2c7c0bb0d213aee5ac4cb0ed3f2b674578ca752483e WatchSource:0}: Error finding container 555c51faa372df7e73d4c2c7c0bb0d213aee5ac4cb0ed3f2b674578ca752483e: Status 404 returned error can't find the container with id 555c51faa372df7e73d4c2c7c0bb0d213aee5ac4cb0ed3f2b674578ca752483e
Nov 02 15:24:17.548338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:17.548250    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd" event={"ID":"f7831b98-9674-4352-8847-f1119617ae99","Type":"ContainerStarted","Data":"555c51faa372df7e73d4c2c7c0bb0d213aee5ac4cb0ed3f2b674578ca752483e"}
Nov 02 15:24:18.582115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:18.582073    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd" event={"ID":"f7831b98-9674-4352-8847-f1119617ae99","Type":"ContainerStarted","Data":"be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db"}
Nov 02 15:24:18.582703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:18.582677    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:18.622603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:18.622552    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd" podStartSLOduration=2.291415232 podStartE2EDuration="2.622539218s" totalImagesPullingTime="331.123986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:18.620251427 +0000 UTC m=+1710.878320874" watchObservedRunningTime="2026-11-02 15:24:18.622539218 +0000 UTC m=+1710.880608667"
Nov 02 15:24:24.596956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:24.596921    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:24:36.772197 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:36.772157    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:24:50.485972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:50.485928    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"]
Nov 02 15:24:50.486674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:50.486138    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api" containerID="cri-o://502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39" gracePeriod=30
Nov 02 15:24:51.060453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.055806    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:24:51.147672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.121878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs\") pod \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") "
Nov 02 15:24:51.147672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.121983    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca\") pod \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") "
Nov 02 15:24:51.147672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.122068    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j\" (UniqueName: \"kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j\") pod \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") "
Nov 02 15:24:51.147672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.122200    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls\") pod \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\" (UID: \"45ac4dee-61fe-4c9d-8e2e-09ba70976487\") "
Nov 02 15:24:51.147672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.125717    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca" pod "45ac4dee-61fe-4c9d-8e2e-09ba70976487" (UID: "45ac4dee-61fe-4c9d-8e2e-09ba70976487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:51.152846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.152780    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j" pod "45ac4dee-61fe-4c9d-8e2e-09ba70976487" (UID: "45ac4dee-61fe-4c9d-8e2e-09ba70976487"). InnerVolumeSpecName "kube-api-access-xmw4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:51.157004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.156848    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls" pod "45ac4dee-61fe-4c9d-8e2e-09ba70976487" (UID: "45ac4dee-61fe-4c9d-8e2e-09ba70976487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:24:51.170058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.170030    5041 generic.go:356] "Generic (PLEG): container finished" podID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerID="502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39" exitCode=0
Nov 02 15:24:51.171236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.170274    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" event={"ID":"45ac4dee-61fe-4c9d-8e2e-09ba70976487","Type":"ContainerDied","Data":"502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39"}
Nov 02 15:24:51.171236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.170335    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p" event={"ID":"45ac4dee-61fe-4c9d-8e2e-09ba70976487","Type":"ContainerDied","Data":"b058d2fb40031c029ac399745ca06d8d4be1c51d99422cbd8c7e2f222caebf88"}
Nov 02 15:24:51.171236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.170392    5041 scope.go:122] "RemoveContainer" containerID="502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39"
Nov 02 15:24:51.171236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.170643    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"
Nov 02 15:24:51.187873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.187836    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs" pod "45ac4dee-61fe-4c9d-8e2e-09ba70976487" (UID: "45ac4dee-61fe-4c9d-8e2e-09ba70976487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:24:51.194342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.191507    5041 scope.go:122] "RemoveContainer" containerID="ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"
Nov 02 15:24:51.214324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.214307    5041 scope.go:122] "RemoveContainer" containerID="502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39"
Nov 02 15:24:51.214712 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:51.214685    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39\": container with ID starting with 502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39 not found: ID does not exist" containerID="502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39"
Nov 02 15:24:51.214837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.214716    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39"} err="failed to get container status \"502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39\": rpc error: code = NotFound desc = could not find container \"502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39\": container with ID starting with 502d0e5a62b9daae059e161e4a1b28b0fb210d28d6cc1e9ee43541fdf8bced39 not found: ID does not exist"
Nov 02 15:24:51.214837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.214735    5041 scope.go:122] "RemoveContainer" containerID="ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"
Nov 02 15:24:51.215019 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:24:51.214998    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8\": container with ID starting with ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8 not found: ID does not exist" containerID="ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"
Nov 02 15:24:51.215096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.215029    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8"} err="failed to get container status \"ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8\": rpc error: code = NotFound desc = could not find container \"ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8\": container with ID starting with ff639b0d8e6ad43da114a7cc71aa6edde0c708ca132a189f758e4b16f69289a8 not found: ID does not exist"
Nov 02 15:24:51.224939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.224910    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:51.224939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.224936    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ac4dee-61fe-4c9d-8e2e-09ba70976487-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:51.225120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.224948    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmw4j\" (UniqueName: \"kubernetes.io/projected/45ac4dee-61fe-4c9d-8e2e-09ba70976487-kube-api-access-xmw4j\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:51.225120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.224969    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ac4dee-61fe-4c9d-8e2e-09ba70976487-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:24:51.587621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.587591    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"]
Nov 02 15:24:51.596812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.596781    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1cdbe6-7c85c4bdb8-rv95p"]
Nov 02 15:24:51.890565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:24:51.890486    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" path="/var/lib/kubelet/pods/45ac4dee-61fe-4c9d-8e2e-09ba70976487/volumes"
Nov 02 15:25:06.772270 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:25:06.772220    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:25:18.244527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.244490    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"]
Nov 02 15:25:18.245138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.244732    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz" podUID="646b75bc-7edd-46cb-bc61-125e95b3598a" containerName="maas-api" containerID="cri-o://dd4188ccc5eddedb9af047b90e672d2a6bfbb0025000453872e39e1dc9264529" gracePeriod=30
Nov 02 15:25:18.681757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.681694    5041 generic.go:356] "Generic (PLEG): container finished" podID="646b75bc-7edd-46cb-bc61-125e95b3598a" containerID="dd4188ccc5eddedb9af047b90e672d2a6bfbb0025000453872e39e1dc9264529" exitCode=0
Nov 02 15:25:18.682044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.682007    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz" event={"ID":"646b75bc-7edd-46cb-bc61-125e95b3598a","Type":"ContainerDied","Data":"dd4188ccc5eddedb9af047b90e672d2a6bfbb0025000453872e39e1dc9264529"}
Nov 02 15:25:18.702636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.702599    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.841759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c\" (UniqueName: \"kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c\") pod \"646b75bc-7edd-46cb-bc61-125e95b3598a\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") "
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.841814    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls\") pod \"646b75bc-7edd-46cb-bc61-125e95b3598a\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") "
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.842014    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca\") pod \"646b75bc-7edd-46cb-bc61-125e95b3598a\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") "
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.842073    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs\") pod \"646b75bc-7edd-46cb-bc61-125e95b3598a\" (UID: \"646b75bc-7edd-46cb-bc61-125e95b3598a\") "
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.850849    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls" pod "646b75bc-7edd-46cb-bc61-125e95b3598a" (UID: "646b75bc-7edd-46cb-bc61-125e95b3598a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.853630    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c" pod "646b75bc-7edd-46cb-bc61-125e95b3598a" (UID: "646b75bc-7edd-46cb-bc61-125e95b3598a"). InnerVolumeSpecName "kube-api-access-slv6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:18.877603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.855753    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca" pod "646b75bc-7edd-46cb-bc61-125e95b3598a" (UID: "646b75bc-7edd-46cb-bc61-125e95b3598a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:18.935620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.935564    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs" pod "646b75bc-7edd-46cb-bc61-125e95b3598a" (UID: "646b75bc-7edd-46cb-bc61-125e95b3598a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:18.950139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.948955    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646b75bc-7edd-46cb-bc61-125e95b3598a-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:25:18.950139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.948985    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:25:18.950139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.949000    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv6c\" (UniqueName: \"kubernetes.io/projected/646b75bc-7edd-46cb-bc61-125e95b3598a-kube-api-access-slv6c\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:25:18.950139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:18.949013    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646b75bc-7edd-46cb-bc61-125e95b3598a-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:25:19.691936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:19.691904    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"
Nov 02 15:25:19.691936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:19.691930    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz" event={"ID":"646b75bc-7edd-46cb-bc61-125e95b3598a","Type":"ContainerDied","Data":"5dab8d5b20a80dc415f043090f6e9c1fe5a2b7a30d549b875ea17b0541b44da8"}
Nov 02 15:25:19.692535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:19.691984    5041 scope.go:122] "RemoveContainer" containerID="dd4188ccc5eddedb9af047b90e672d2a6bfbb0025000453872e39e1dc9264529"
Nov 02 15:25:19.739891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:19.739864    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"]
Nov 02 15:25:19.749197 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:19.749174    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-815746-5d7c94ff9f-v7lxz"]
Nov 02 15:25:19.890439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:19.890407    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646b75bc-7edd-46cb-bc61-125e95b3598a" path="/var/lib/kubelet/pods/646b75bc-7edd-46cb-bc61-125e95b3598a/volumes"
Nov 02 15:25:34.558479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.558443    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"]
Nov 02 15:25:34.559410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559391    5041 cpu_manager.go:414] "containerMap: removing container" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api"
Nov 02 15:25:34.559410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559410    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api"
Nov 02 15:25:34.559538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559428    5041 cpu_manager.go:414] "containerMap: removing container" podUID="646b75bc-7edd-46cb-bc61-125e95b3598a" containerName="maas-api"
Nov 02 15:25:34.559538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559434    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646b75bc-7edd-46cb-bc61-125e95b3598a" containerName="maas-api"
Nov 02 15:25:34.559615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559573    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api"
Nov 02 15:25:34.559615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559583    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="646b75bc-7edd-46cb-bc61-125e95b3598a" containerName="maas-api"
Nov 02 15:25:34.559615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.559590    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api"
Nov 02 15:25:34.565302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.565277    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.598905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.598880    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2f8b7c\""
Nov 02 15:25:34.599021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.598964    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2f8b7c-dockercfg-5fn6z\""
Nov 02 15:25:34.603907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.603881    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"]
Nov 02 15:25:34.756387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.710191    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflgl\" (UniqueName: \"kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl\") pod \"payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.756387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.710430    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.837846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.836521    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.840187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.838343    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflgl\" (UniqueName: \"kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl\") pod \"payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.844338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.844303    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.883253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.870569    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"]
Nov 02 15:25:34.883253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.872073    5041 cpu_manager.go:414] "containerMap: removing container" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api"
Nov 02 15:25:34.883253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.872098    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ac4dee-61fe-4c9d-8e2e-09ba70976487" containerName="maas-api"
Nov 02 15:25:34.897024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.896600    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:34.897927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.897405    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflgl\" (UniqueName: \"kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl\") pod \"payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.897927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.897811    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:34.900600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:34.900578    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"]
Nov 02 15:25:35.043578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.043540    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g722\" (UniqueName: \"kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722\") pod \"payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.043765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.043666    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.145056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.144971    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g722\" (UniqueName: \"kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722\") pod \"payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.145213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.145064    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.145805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.145783    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.168229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.168204    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g722\" (UniqueName: \"kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722\") pod \"payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.233952 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:25:35.233918    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e4fb40_3326_425b_be8c_40c03199cf10.slice/crio-f4d4e88a4b113ad21bcd78de07687073528461e2b703d730c8ceac8ba00a26ff WatchSource:0}: Error finding container f4d4e88a4b113ad21bcd78de07687073528461e2b703d730c8ceac8ba00a26ff: Status 404 returned error can't find the container with id f4d4e88a4b113ad21bcd78de07687073528461e2b703d730c8ceac8ba00a26ff
Nov 02 15:25:35.234229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.234208    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"]
Nov 02 15:25:35.266993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.266190    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:35.581446 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:25:35.581415    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a621c27_1503_48ee_a17b_f25f9d4f0b3a.slice/crio-7e764e6056e6bbcca1f0c5ce9a415bfd55f2f50966aeb728e530eb1a275a59fa WatchSource:0}: Error finding container 7e764e6056e6bbcca1f0c5ce9a415bfd55f2f50966aeb728e530eb1a275a59fa: Status 404 returned error can't find the container with id 7e764e6056e6bbcca1f0c5ce9a415bfd55f2f50966aeb728e530eb1a275a59fa
Nov 02 15:25:35.581790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:35.581731    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"]
Nov 02 15:25:36.216007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.215969    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" event={"ID":"31e4fb40-3326-425b-be8c-40c03199cf10","Type":"ContainerStarted","Data":"ab6cd1a2c63f9da88223746aa5dc123918f108eb9b0d59362a919d4a15f32f16"}
Nov 02 15:25:36.216209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.216032    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:36.216209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.216058    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" event={"ID":"31e4fb40-3326-425b-be8c-40c03199cf10","Type":"ContainerStarted","Data":"f4d4e88a4b113ad21bcd78de07687073528461e2b703d730c8ceac8ba00a26ff"}
Nov 02 15:25:36.217663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.217633    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" event={"ID":"4a621c27-1503-48ee-a17b-f25f9d4f0b3a","Type":"ContainerStarted","Data":"a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae"}
Nov 02 15:25:36.217776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.217685    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" event={"ID":"4a621c27-1503-48ee-a17b-f25f9d4f0b3a","Type":"ContainerStarted","Data":"7e764e6056e6bbcca1f0c5ce9a415bfd55f2f50966aeb728e530eb1a275a59fa"}
Nov 02 15:25:36.217776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.217713    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:36.252032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.251978    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" podStartSLOduration=2.251965356 podStartE2EDuration="2.251965356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:36.251205642 +0000 UTC m=+1788.509275092" watchObservedRunningTime="2026-11-02 15:25:36.251965356 +0000 UTC m=+1788.510034806"
Nov 02 15:25:36.291413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:36.291371    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" podStartSLOduration=2.291335519 podStartE2EDuration="2.291335519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:36.28866208 +0000 UTC m=+1788.546731534" watchObservedRunningTime="2026-11-02 15:25:36.291335519 +0000 UTC m=+1788.549405009"
Nov 02 15:25:36.819167 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:25:36.819125    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:25:44.810743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:44.810711    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"]
Nov 02 15:25:44.816049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:44.816026    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:44.848003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:44.847974    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"]
Nov 02 15:25:44.849904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:44.849882    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-28f621-dockercfg-7cvsd\""
Nov 02 15:25:44.849904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:44.849889    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-28f621\""
Nov 02 15:25:45.156121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.156018    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.159054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.159025    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgcp\" (UniqueName: \"kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp\") pod \"payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.261471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.261433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.261650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.261517    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgcp\" (UniqueName: \"kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp\") pod \"payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.262104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.262083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.306868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.306841    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgcp\" (UniqueName: \"kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp\") pod \"payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.434059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.434027    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:45.745326 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:25:45.745294    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bf93b2_7a33_4e17_9b79_53328a28761d.slice/crio-1d8510baedf6ca0262bd4a241d5b8bfa8f0b8051479b5fe1f1ec21499467b1ab WatchSource:0}: Error finding container 1d8510baedf6ca0262bd4a241d5b8bfa8f0b8051479b5fe1f1ec21499467b1ab: Status 404 returned error can't find the container with id 1d8510baedf6ca0262bd4a241d5b8bfa8f0b8051479b5fe1f1ec21499467b1ab
Nov 02 15:25:45.746352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:45.746330    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"]
Nov 02 15:25:46.352682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:46.352631    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" event={"ID":"f4bf93b2-7a33-4e17-9b79-53328a28761d","Type":"ContainerStarted","Data":"8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0"}
Nov 02 15:25:46.353117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:46.352697    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" event={"ID":"f4bf93b2-7a33-4e17-9b79-53328a28761d","Type":"ContainerStarted","Data":"1d8510baedf6ca0262bd4a241d5b8bfa8f0b8051479b5fe1f1ec21499467b1ab"}
Nov 02 15:25:46.353117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:46.352724    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:25:46.392013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:46.391958    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" podStartSLOduration=2.391946466 podStartE2EDuration="2.391946466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:46.388125782 +0000 UTC m=+1798.646195231" watchObservedRunningTime="2026-11-02 15:25:46.391946466 +0000 UTC m=+1798.650015915"
Nov 02 15:25:47.270407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:47.270349    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:25:47.283222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:47.283191    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:25:48.922525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:48.922503    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:25:48.932836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:48.932808    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:25:57.366237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:25:57.366207    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:26:06.770136 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:26:06.770099    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:26:09.234654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.234614    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"]
Nov 02 15:26:09.240443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.240420    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.258548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.258504    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2f8b7c-serving-cert\""
Nov 02 15:26:09.273488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.273461    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"]
Nov 02 15:26:09.346498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.346462    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.346661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.346626    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtbwk\" (UniqueName: \"kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.346784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.346765    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.346849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.346831    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.451394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.449522    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.451394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.449637    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.451394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.449770    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.451394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.449935    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbwk\" (UniqueName: \"kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.452224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.452186    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.469043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.469009    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.480553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.480232    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.484446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.483394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbwk\" (UniqueName: \"kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk\") pod \"maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.559507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.559423    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:09.879814 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:26:09.879782    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734bf2eb_db6f_4808_b21e_6d1bfb141987.slice/crio-fc50abb84d3c1fc9129935ea6585027b3fa61a4b74937de3c57dd783948167f3 WatchSource:0}: Error finding container fc50abb84d3c1fc9129935ea6585027b3fa61a4b74937de3c57dd783948167f3: Status 404 returned error can't find the container with id fc50abb84d3c1fc9129935ea6585027b3fa61a4b74937de3c57dd783948167f3
Nov 02 15:26:09.881849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.881824    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:26:09.893756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:09.893722    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"]
Nov 02 15:26:10.569644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:10.569602    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c" event={"ID":"734bf2eb-db6f-4808-b21e-6d1bfb141987","Type":"ContainerStarted","Data":"141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1"}
Nov 02 15:26:10.570066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:10.569659    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c" event={"ID":"734bf2eb-db6f-4808-b21e-6d1bfb141987","Type":"ContainerStarted","Data":"fc50abb84d3c1fc9129935ea6585027b3fa61a4b74937de3c57dd783948167f3"}
Nov 02 15:26:10.570066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:10.569711    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:10.611751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:10.611658    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c" podStartSLOduration=1.17123621 podStartE2EDuration="1.611645106s" totalImagesPullingTime="440.408896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:10.606615191 +0000 UTC m=+1822.864684642" watchObservedRunningTime="2026-11-02 15:26:10.611645106 +0000 UTC m=+1822.869714555"
Nov 02 15:26:16.041264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.041227    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"]
Nov 02 15:26:16.041806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.041503    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc" podUID="267d929a-319e-4acc-8494-83f1b42cedd1" containerName="maas-api" containerID="cri-o://c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75" gracePeriod=30
Nov 02 15:26:16.574611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.574581    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:26:16.613412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.612540    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:26:16.680351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.680312    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs\") pod \"267d929a-319e-4acc-8494-83f1b42cedd1\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") "
Nov 02 15:26:16.691385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.680674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6\" (UniqueName: \"kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6\") pod \"267d929a-319e-4acc-8494-83f1b42cedd1\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") "
Nov 02 15:26:16.691385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.687611    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls\") pod \"267d929a-319e-4acc-8494-83f1b42cedd1\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") "
Nov 02 15:26:16.691385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.687824    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca\") pod \"267d929a-319e-4acc-8494-83f1b42cedd1\" (UID: \"267d929a-319e-4acc-8494-83f1b42cedd1\") "
Nov 02 15:26:16.697812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.697778    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca" pod "267d929a-319e-4acc-8494-83f1b42cedd1" (UID: "267d929a-319e-4acc-8494-83f1b42cedd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:26:16.703068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.702569    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs" pod "267d929a-319e-4acc-8494-83f1b42cedd1" (UID: "267d929a-319e-4acc-8494-83f1b42cedd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:26:16.703068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.702722    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6" pod "267d929a-319e-4acc-8494-83f1b42cedd1" (UID: "267d929a-319e-4acc-8494-83f1b42cedd1"). InnerVolumeSpecName "kube-api-access-4bsw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:16.703272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.703115    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls" pod "267d929a-319e-4acc-8494-83f1b42cedd1" (UID: "267d929a-319e-4acc-8494-83f1b42cedd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:26:16.712138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.712109    5041 generic.go:356] "Generic (PLEG): container finished" podID="267d929a-319e-4acc-8494-83f1b42cedd1" containerID="c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75" exitCode=0
Nov 02 15:26:16.712304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.712160    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"
Nov 02 15:26:16.712545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.712420    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc" event={"ID":"267d929a-319e-4acc-8494-83f1b42cedd1","Type":"ContainerDied","Data":"c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75"}
Nov 02 15:26:16.712545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.712477    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc" event={"ID":"267d929a-319e-4acc-8494-83f1b42cedd1","Type":"ContainerDied","Data":"e85015d591e7a0909cc5d6e4b4ca11ec178d98d4969e4fd9d406c945398b3567"}
Nov 02 15:26:16.712545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.712508    5041 scope.go:122] "RemoveContainer" containerID="c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75"
Nov 02 15:26:16.750963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.750943    5041 scope.go:122] "RemoveContainer" containerID="c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75"
Nov 02 15:26:16.751329 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:26:16.751298    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75\": container with ID starting with c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75 not found: ID does not exist" containerID="c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75"
Nov 02 15:26:16.751423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.751343    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75"} err="failed to get container status \"c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75\": rpc error: code = NotFound desc = could not find container \"c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75\": container with ID starting with c6d1fc0171353a02dfefd6195c25475b4f7fd4f786d6051f2b79410ace937c75 not found: ID does not exist"
Nov 02 15:26:16.780650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.780621    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"]
Nov 02 15:26:16.798318 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.798282    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec02d-845df4848-ndgvc"]
Nov 02 15:26:16.798703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.798676    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:26:16.798827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.798709    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267d929a-319e-4acc-8494-83f1b42cedd1-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:26:16.798827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.798720    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267d929a-319e-4acc-8494-83f1b42cedd1-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:26:16.798827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:16.798728    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bsw6\" (UniqueName: \"kubernetes.io/projected/267d929a-319e-4acc-8494-83f1b42cedd1-kube-api-access-4bsw6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:26:17.893546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:17.893512    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267d929a-319e-4acc-8494-83f1b42cedd1" path="/var/lib/kubelet/pods/267d929a-319e-4acc-8494-83f1b42cedd1/volumes"
Nov 02 15:26:20.232138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.232099    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"]
Nov 02 15:26:20.233426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.233399    5041 cpu_manager.go:414] "containerMap: removing container" podUID="267d929a-319e-4acc-8494-83f1b42cedd1" containerName="maas-api"
Nov 02 15:26:20.233426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.233423    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267d929a-319e-4acc-8494-83f1b42cedd1" containerName="maas-api"
Nov 02 15:26:20.233656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.233641    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="267d929a-319e-4acc-8494-83f1b42cedd1" containerName="maas-api"
Nov 02 15:26:20.239321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.239302    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.264151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.264131    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-28f621-serving-cert\""
Nov 02 15:26:20.279396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.272743    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"]
Nov 02 15:26:20.291897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.291869    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdgw\" (UniqueName: \"kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.292203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.292178    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.292290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.292233    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.292454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.292425    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.393853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.393827    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.393984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.393880    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.393984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.393913    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.393984 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:26:20.393954    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-serving-cert: secret "maas-api-e2e-inf-a-28f621-serving-cert" not found
Nov 02 15:26:20.394153 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:26:20.394014    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls podName:e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e nodeName:}" failed. No retries permitted until 2026-11-02 15:26:20.893993214 +0000 UTC m=+1833.152062650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls") pod "maas-api-e2e-inf-a-28f621-7857694587-d8zj6" (UID: "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e") : secret "maas-api-e2e-inf-a-28f621-serving-cert" not found
Nov 02 15:26:20.394153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.394036    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdgw\" (UniqueName: \"kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.394618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.394598    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.396558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.396536    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.421384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.418418    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdgw\" (UniqueName: \"kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.909489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.909449    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:20.911980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:20.911956    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-28f621-7857694587-d8zj6\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:21.164786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:21.164693    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:21.590745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:21.590623    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"]
Nov 02 15:26:21.806968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:21.806930    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6" event={"ID":"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e","Type":"ContainerStarted","Data":"6ffa20cbad9dab01d16bb14275047901c76e26419f85c132e7f64e025ca736f4"}
Nov 02 15:26:22.818293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:22.818253    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6" event={"ID":"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e","Type":"ContainerStarted","Data":"27e4cb46196784daccee9cf34af3ba7ac98df585a5da9648588916a60a4ff5a9"}
Nov 02 15:26:22.818715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:22.818405    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:22.856900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:22.856845    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6" podStartSLOduration=2.47106137 podStartE2EDuration="2.856833185s" totalImagesPullingTime="385.771815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:22.855979143 +0000 UTC m=+1835.114048593" watchObservedRunningTime="2026-11-02 15:26:22.856833185 +0000 UTC m=+1835.114902635"
Nov 02 15:26:28.833118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:28.833037    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:26:33.450402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.450347    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"]
Nov 02 15:26:33.456339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.456308    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:33.473610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.473590    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-80ec03-dockercfg-jzh4r\""
Nov 02 15:26:33.483486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.483460    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-80ec03\""
Nov 02 15:26:33.488532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.488509    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"]
Nov 02 15:26:33.784024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.783931    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:33.784187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.784080    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5kc\" (UniqueName: \"kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc\") pod \"payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:33.885346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.885310    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5kc\" (UniqueName: \"kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc\") pod \"payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:33.885589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.885567    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:33.886304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.886282    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:33.911901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:33.911875    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5kc\" (UniqueName: \"kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc\") pod \"payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:34.071346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:34.071280    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:34.383180 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:26:34.383147    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738bb0ae_3436_4de8_9cbf_30a55e30a104.slice/crio-b6b001008fd6d01b21888caa4fdeab24f18503c6ca956115578b8962c9b6d262 WatchSource:0}: Error finding container b6b001008fd6d01b21888caa4fdeab24f18503c6ca956115578b8962c9b6d262: Status 404 returned error can't find the container with id b6b001008fd6d01b21888caa4fdeab24f18503c6ca956115578b8962c9b6d262
Nov 02 15:26:34.383397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:34.383344    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"]
Nov 02 15:26:34.944923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:34.944880    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" event={"ID":"738bb0ae-3436-4de8-9cbf-30a55e30a104","Type":"ContainerStarted","Data":"853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f"}
Nov 02 15:26:34.945480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:34.944950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" event={"ID":"738bb0ae-3436-4de8-9cbf-30a55e30a104","Type":"ContainerStarted","Data":"b6b001008fd6d01b21888caa4fdeab24f18503c6ca956115578b8962c9b6d262"}
Nov 02 15:26:34.945480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:34.945033    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:34.982035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:34.981988    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" podStartSLOduration=1.981977204 podStartE2EDuration="1.981977204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:34.98015971 +0000 UTC m=+1847.238229160" watchObservedRunningTime="2026-11-02 15:26:34.981977204 +0000 UTC m=+1847.240046653"
Nov 02 15:26:36.771130 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:26:36.771093    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:26:45.956753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:45.956723    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:26:54.050331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.050296    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"]
Nov 02 15:26:54.059348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.059320    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.075175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.075144    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"]
Nov 02 15:26:54.078257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.078229    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-69ff88\""
Nov 02 15:26:54.078257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.078245    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-69ff88-dockercfg-kts45\""
Nov 02 15:26:54.257217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.257175    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.258520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.258487    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmvg\" (UniqueName: \"kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg\") pod \"payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.369010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.368898    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.369421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.369389    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmvg\" (UniqueName: \"kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg\") pod \"payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.370800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.370769    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.409090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.409050    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmvg\" (UniqueName: \"kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg\") pod \"payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.683311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.683280    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:54.989492 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:26:54.989461    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11016326_7fb1_43f2_8377_007942c7fbe7.slice/crio-11ac1633adbcec866135eac777d0823842834ced761b7cd54dcc889de7fb170b WatchSource:0}: Error finding container 11ac1633adbcec866135eac777d0823842834ced761b7cd54dcc889de7fb170b: Status 404 returned error can't find the container with id 11ac1633adbcec866135eac777d0823842834ced761b7cd54dcc889de7fb170b
Nov 02 15:26:54.990343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:54.990305    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"]
Nov 02 15:26:55.431424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:55.431318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" event={"ID":"11016326-7fb1-43f2-8377-007942c7fbe7","Type":"ContainerStarted","Data":"6e75610bcef7c1b867aa8e3d76b053448acebd593bece6fa98f316e350c0c868"}
Nov 02 15:26:55.431424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:55.431414    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" event={"ID":"11016326-7fb1-43f2-8377-007942c7fbe7","Type":"ContainerStarted","Data":"11ac1633adbcec866135eac777d0823842834ced761b7cd54dcc889de7fb170b"}
Nov 02 15:26:55.432864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:55.432621    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:26:55.490912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:26:55.490852    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" podStartSLOduration=1.490838018 podStartE2EDuration="1.490838018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:55.487599968 +0000 UTC m=+1867.745669419" watchObservedRunningTime="2026-11-02 15:26:55.490838018 +0000 UTC m=+1867.748907467"
Nov 02 15:27:06.444451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:06.444415    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:27:06.769644 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:27:06.769553    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:27:12.754565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:12.754530    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"]
Nov 02 15:27:12.755000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:12.754726    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" containerID="cri-o://220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384" gracePeriod=30
Nov 02 15:27:15.274778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.274743    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"]
Nov 02 15:27:15.279360 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.279336    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.310212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.310182    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"]
Nov 02 15:27:15.358688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.358657    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-80ec03-serving-cert\""
Nov 02 15:27:15.373683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.373652    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.373922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.373833    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.374077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.374054    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.374350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.374331    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjp24\" (UniqueName: \"kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.475449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.475421    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.475616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.475475    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.475616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.475529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.475746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.475734    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjp24\" (UniqueName: \"kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.476224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.476187    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.477707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.477678    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.477865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.477845    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.497406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.497380    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjp24\" (UniqueName: \"kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24\") pod \"maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.611292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.611204    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:15.955276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:15.955243    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"]
Nov 02 15:27:15.955962 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:27:15.955937    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ad83f6_7c13_4908_b2cb_9ffc34f39ce5.slice/crio-a0f3eb4f184d855897bad0fb4679590d2f7b1688f2bf5772d9dcf4d55fe33078 WatchSource:0}: Error finding container a0f3eb4f184d855897bad0fb4679590d2f7b1688f2bf5772d9dcf4d55fe33078: Status 404 returned error can't find the container with id a0f3eb4f184d855897bad0fb4679590d2f7b1688f2bf5772d9dcf4d55fe33078
Nov 02 15:27:16.723295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.723262    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"]
Nov 02 15:27:16.727671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.727646    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:16.744792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.744763    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:27:16.744947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.744812    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:27:16.745407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.745386    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cfc63a\""
Nov 02 15:27:16.754681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.754655    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cfc63a-dockercfg-tl5z6\""
Nov 02 15:27:16.769147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.769107    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"]
Nov 02 15:27:16.774417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.774348    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj" event={"ID":"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5","Type":"ContainerStarted","Data":"be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b"}
Nov 02 15:27:16.774542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.774447    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj" event={"ID":"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5","Type":"ContainerStarted","Data":"a0f3eb4f184d855897bad0fb4679590d2f7b1688f2bf5772d9dcf4d55fe33078"}
Nov 02 15:27:16.774542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.774493    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:16.826848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.826818    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64545\" (UniqueName: \"kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545\") pod \"payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:16.827141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.827109    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:16.828672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.828618    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj" podStartSLOduration=1.465167038 podStartE2EDuration="1.828602644s" totalImagesPullingTime="363.435606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:16.816626869 +0000 UTC m=+1889.074696336" watchObservedRunningTime="2026-11-02 15:27:16.828602644 +0000 UTC m=+1889.086672096"
Nov 02 15:27:16.855921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.855729    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"]
Nov 02 15:27:16.861327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.861304    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:16.892657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.892627    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"]
Nov 02 15:27:16.928343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.928311    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:16.928547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.928354    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:16.928622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.928560    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqbq\" (UniqueName: \"kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq\") pod \"payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:16.928686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.928622    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64545\" (UniqueName: \"kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545\") pod \"payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:16.929662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:16.929637    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:17.042442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.030324    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:17.042442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.030670    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqbq\" (UniqueName: \"kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq\") pod \"payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:17.207418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.200831    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:17.207418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.202994    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64545\" (UniqueName: \"kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545\") pod \"payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:17.241969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.240734    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqbq\" (UniqueName: \"kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq\") pod \"payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:17.306074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.304649    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:17.348991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.348963    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:17.809968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.809929    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"]
Nov 02 15:27:17.856507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.856466    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" event={"ID":"687dc132-d1be-4d39-aa20-873e5a4191f0","Type":"ContainerStarted","Data":"0ccd8e02d754eb764dca5185a997732e22cbbde76d7a706a240f40d382511241"}
Nov 02 15:27:17.904391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:17.903962    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"]
Nov 02 15:27:17.908417 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:27:17.908388    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75824f88_123f_4256_8369_ccb1efcbaa50.slice/crio-9dda5a5ca41c519e19f0a89a0db9410e0818b2979ac616e6ad56fa9ab71d3fbe WatchSource:0}: Error finding container 9dda5a5ca41c519e19f0a89a0db9410e0818b2979ac616e6ad56fa9ab71d3fbe: Status 404 returned error can't find the container with id 9dda5a5ca41c519e19f0a89a0db9410e0818b2979ac616e6ad56fa9ab71d3fbe
Nov 02 15:27:18.125553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.125523    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:27:18.232105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232069    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232199    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232225    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232251    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232273    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232414    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232481    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.232583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.232564    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket\") pod \"823c93a5-c32c-413e-ae47-392e56b40af4\" (UID: \"823c93a5-c32c-413e-ae47-392e56b40af4\") "
Nov 02 15:27:18.233238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.233212    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.233512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.233486    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:18.234768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.234001    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.234960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.234921    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.235114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.235095    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.237755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.237730    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:18.237874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.237821    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:18.238081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.238055    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "kube-api-access-smwpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:18.239905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.239879    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy" pod "823c93a5-c32c-413e-ae47-392e56b40af4" (UID: "823c93a5-c32c-413e-ae47-392e56b40af4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.334620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334587    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwpd\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-kube-api-access-smwpd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334619    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334629    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823c93a5-c32c-413e-ae47-392e56b40af4-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334638    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334646    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334654    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334666    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823c93a5-c32c-413e-ae47-392e56b40af4-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334679    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823c93a5-c32c-413e-ae47-392e56b40af4-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.334843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.334692    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823c93a5-c32c-413e-ae47-392e56b40af4-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.867671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.867579    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" event={"ID":"687dc132-d1be-4d39-aa20-873e5a4191f0","Type":"ContainerStarted","Data":"2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e"}
Nov 02 15:27:18.868131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.867679    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:18.869399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.869338    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" event={"ID":"75824f88-123f-4256-8369-ccb1efcbaa50","Type":"ContainerStarted","Data":"4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906"}
Nov 02 15:27:18.869537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.869412    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" event={"ID":"75824f88-123f-4256-8369-ccb1efcbaa50","Type":"ContainerStarted","Data":"9dda5a5ca41c519e19f0a89a0db9410e0818b2979ac616e6ad56fa9ab71d3fbe"}
Nov 02 15:27:18.869537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.869439    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:18.870778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.870757    5041 generic.go:356] "Generic (PLEG): container finished" podID="823c93a5-c32c-413e-ae47-392e56b40af4" containerID="220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384" exitCode=0
Nov 02 15:27:18.870868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.870780    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" event={"ID":"823c93a5-c32c-413e-ae47-392e56b40af4","Type":"ContainerDied","Data":"220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384"}
Nov 02 15:27:18.870868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.870797    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"
Nov 02 15:27:18.870868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.870832    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt" event={"ID":"823c93a5-c32c-413e-ae47-392e56b40af4","Type":"ContainerDied","Data":"35d61e9fdd301baef01a700abf698e4b68fc4bb1adbd42e4d4c64a611e73d610"}
Nov 02 15:27:18.870868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.870866    5041 scope.go:122] "RemoveContainer" containerID="220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384"
Nov 02 15:27:18.884158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.884133    5041 scope.go:122] "RemoveContainer" containerID="220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384"
Nov 02 15:27:18.884498 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:27:18.884468    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384\": container with ID starting with 220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384 not found: ID does not exist" containerID="220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384"
Nov 02 15:27:18.884598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.884504    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384"} err="failed to get container status \"220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384\": rpc error: code = NotFound desc = could not find container \"220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384\": container with ID starting with 220c0bff485d4989215f2a0827864e1b72784e4ac2ee016b70e58e32b9251384 not found: ID does not exist"
Nov 02 15:27:18.907759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.907721    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" podStartSLOduration=2.907702698 podStartE2EDuration="2.907702698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:18.902748386 +0000 UTC m=+1891.160817848" watchObservedRunningTime="2026-11-02 15:27:18.907702698 +0000 UTC m=+1891.165772164"
Nov 02 15:27:18.948049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.948004    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" podStartSLOduration=2.94798923 podStartE2EDuration="2.94798923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:18.94270845 +0000 UTC m=+1891.200777901" watchObservedRunningTime="2026-11-02 15:27:18.94798923 +0000 UTC m=+1891.206058684"
Nov 02 15:27:18.979408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.979384    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"]
Nov 02 15:27:18.988409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:18.988389    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-9vcdt"]
Nov 02 15:27:19.889897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:19.889865    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" path="/var/lib/kubelet/pods/823c93a5-c32c-413e-ae47-392e56b40af4/volumes"
Nov 02 15:27:21.714417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:21.714351    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"]
Nov 02 15:27:21.714862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:21.714634    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" containerID="cri-o://23706eb38ed8df0a9c02d35f98b40a3609f53bffbedd768575286f961dc2eb33" gracePeriod=30
Nov 02 15:27:22.866181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:22.866149    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:27:25.691153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:25.691116    5041 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:27:25.691556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:25.691175    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:27:26.990680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:26.990651    5041 generic.go:356] "Generic (PLEG): container finished" podID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerID="23706eb38ed8df0a9c02d35f98b40a3609f53bffbedd768575286f961dc2eb33" exitCode=0
Nov 02 15:27:26.991061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:26.990725    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" event={"ID":"0af9fd91-65d3-486d-9d88-c098e6a85e84","Type":"ContainerDied","Data":"23706eb38ed8df0a9c02d35f98b40a3609f53bffbedd768575286f961dc2eb33"}
Nov 02 15:27:26.991061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:26.990790    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n" event={"ID":"0af9fd91-65d3-486d-9d88-c098e6a85e84","Type":"ContainerDied","Data":"bf8075f62ab3ac39ea7bd77739562245b393adc5fe5173a90154b2163934d4b4"}
Nov 02 15:27:26.991061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:26.990820    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8075f62ab3ac39ea7bd77739562245b393adc5fe5173a90154b2163934d4b4"
Nov 02 15:27:27.023749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.023725    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:27:27.073231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073204    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073271    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073375    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073443    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073484    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:27.073527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073499    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073634    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073660    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073743    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:27.073764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073761    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data\") pod \"0af9fd91-65d3-486d-9d88-c098e6a85e84\" (UID: \"0af9fd91-65d3-486d-9d88-c098e6a85e84\") "
Nov 02 15:27:27.073981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.073918    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:27.074206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.074150    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:27.074515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.074478    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af9fd91-65d3-486d-9d88-c098e6a85e84-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.074515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.074505    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.074700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.074521    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.074700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.074536    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.074700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.074611    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:27.075908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.075887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:27.076059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.076038    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:27.076160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.076141    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "kube-api-access-c5d4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:27.076212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.076146    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo" pod "0af9fd91-65d3-486d-9d88-c098e6a85e84" (UID: "0af9fd91-65d3-486d-9d88-c098e6a85e84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:27.176434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.176403    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.176434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.176431    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.176434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.176440    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.176654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.176449    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5d4k\" (UniqueName: \"kubernetes.io/projected/0af9fd91-65d3-486d-9d88-c098e6a85e84-kube-api-access-c5d4k\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.176654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.176458    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af9fd91-65d3-486d-9d88-c098e6a85e84-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:27.733834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.733795    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"]
Nov 02 15:27:27.734099 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:27.734070    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" containerID="cri-o://dfaa8d1035d9156b5b76609127280b290a3f52e9e0bb5a1bca3d323002403a05" gracePeriod=30
Nov 02 15:27:28.000143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:28.000062    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"
Nov 02 15:27:28.051871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:28.051836    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"]
Nov 02 15:27:28.057037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:28.057007    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-tql4n"]
Nov 02 15:27:29.890443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:29.890408    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" path="/var/lib/kubelet/pods/0af9fd91-65d3-486d-9d88-c098e6a85e84/volumes"
Nov 02 15:27:29.890881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:29.890865    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:27:29.890927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:29.890916    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:27:32.390120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:32.390086    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:27:32.390539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:32.390146    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:27:33.171497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.171452    5041 generic.go:356] "Generic (PLEG): container finished" podID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerID="dfaa8d1035d9156b5b76609127280b290a3f52e9e0bb5a1bca3d323002403a05" exitCode=0
Nov 02 15:27:33.171765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.171562    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" event={"ID":"44b0bd35-d966-45a0-84a2-6d0d492fa1c7","Type":"ContainerDied","Data":"dfaa8d1035d9156b5b76609127280b290a3f52e9e0bb5a1bca3d323002403a05"}
Nov 02 15:27:33.202336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.202309    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:27:33.303470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303436    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.303623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303539    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.303678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303625    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.303678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303668    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.303780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303767    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.303843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303831    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.303904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303768    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:33.303965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303947    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:33.304038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303983    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:33.304038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.303952    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.304144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304041    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.304144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304036    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:33.304144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304092    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket\") pod \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\" (UID: \"44b0bd35-d966-45a0-84a2-6d0d492fa1c7\") "
Nov 02 15:27:33.304343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304324    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:33.304772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304752    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.304877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304778    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.304877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304794    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.304877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304806    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.304877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.304820    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.306334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.306307    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:33.306608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.306584    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:33.306694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.306627    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:33.306694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.306655    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488" pod "44b0bd35-d966-45a0-84a2-6d0d492fa1c7" (UID: "44b0bd35-d966-45a0-84a2-6d0d492fa1c7"). InnerVolumeSpecName "kube-api-access-sh488". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:33.406549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.406516    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.406549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.406542    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.406549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.406552    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh488\" (UniqueName: \"kubernetes.io/projected/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-kube-api-access-sh488\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:33.407008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:33.406561    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b0bd35-d966-45a0-84a2-6d0d492fa1c7-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:27:34.181677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:34.181650    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"
Nov 02 15:27:34.181820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:34.181691    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk" event={"ID":"44b0bd35-d966-45a0-84a2-6d0d492fa1c7","Type":"ContainerDied","Data":"99d89a2c54539e4b73309c4857849c18955f9757dee43b8d2c02097f0adc420e"}
Nov 02 15:27:34.181820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:34.181754    5041 scope.go:122] "RemoveContainer" containerID="dfaa8d1035d9156b5b76609127280b290a3f52e9e0bb5a1bca3d323002403a05"
Nov 02 15:27:34.243260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:34.243230    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"]
Nov 02 15:27:34.249503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:34.249481    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-9mlzk"]
Nov 02 15:27:35.895531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:35.895491    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" path="/var/lib/kubelet/pods/44b0bd35-d966-45a0-84a2-6d0d492fa1c7/volumes"
Nov 02 15:27:36.040991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.040961    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"]
Nov 02 15:27:36.041900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.041882    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy"
Nov 02 15:27:36.041964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.041902    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy"
Nov 02 15:27:36.041964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.041928    5041 cpu_manager.go:414] "containerMap: removing container" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy"
Nov 02 15:27:36.041964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.041933    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy"
Nov 02 15:27:36.041964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.041948    5041 cpu_manager.go:414] "containerMap: removing container" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy"
Nov 02 15:27:36.041964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.041954    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy"
Nov 02 15:27:36.042141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.042118    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af9fd91-65d3-486d-9d88-c098e6a85e84" containerName="istio-proxy"
Nov 02 15:27:36.042173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.042140    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="823c93a5-c32c-413e-ae47-392e56b40af4" containerName="istio-proxy"
Nov 02 15:27:36.042173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.042149    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b0bd35-d966-45a0-84a2-6d0d492fa1c7" containerName="istio-proxy"
Nov 02 15:27:36.047770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.047750    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.065645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.065619    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-69ff88-serving-cert\""
Nov 02 15:27:36.083991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.083571    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"]
Nov 02 15:27:36.143918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.143886    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txr2t\" (UniqueName: \"kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.144064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.144022    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.144166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.144152    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.144232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.144219    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.247446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.247401    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.247661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.247601    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.247744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.247691    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.247852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.247831    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txr2t\" (UniqueName: \"kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.249127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.249098    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.277171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.277123    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.281277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.281240    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.281666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.281640    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txr2t\" (UniqueName: \"kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t\") pod \"maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.364773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.364741    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:36.676337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:36.676299    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"]
Nov 02 15:27:36.676824 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:27:36.676799    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf206e5b3_c341_47ea_ade7_31819dc76c46.slice/crio-84a8105e669e2b19baa094050812fced2d8feebe645d04c43bae799937e5c415 WatchSource:0}: Error finding container 84a8105e669e2b19baa094050812fced2d8feebe645d04c43bae799937e5c415: Status 404 returned error can't find the container with id 84a8105e669e2b19baa094050812fced2d8feebe645d04c43bae799937e5c415
Nov 02 15:27:36.771516 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:27:36.771479    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:27:37.254880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:37.254790    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9" event={"ID":"f206e5b3-c341-47ea-ade7-31819dc76c46","Type":"ContainerStarted","Data":"b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9"}
Nov 02 15:27:37.254880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:37.254858    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9" event={"ID":"f206e5b3-c341-47ea-ade7-31819dc76c46","Type":"ContainerStarted","Data":"84a8105e669e2b19baa094050812fced2d8feebe645d04c43bae799937e5c415"}
Nov 02 15:27:37.255396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:37.254889    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:37.310239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:37.310194    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9" podStartSLOduration=0.99021822 podStartE2EDuration="1.310181132s" totalImagesPullingTime="319.962912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:37.305776954 +0000 UTC m=+1909.563846404" watchObservedRunningTime="2026-11-02 15:27:37.310181132 +0000 UTC m=+1909.568250581"
Nov 02 15:27:43.270081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:43.270045    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:27:51.191522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.191487    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"]
Nov 02 15:27:51.198162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.198135    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.213759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.213734    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d35a86-serving-cert\""
Nov 02 15:27:51.233784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.233742    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"]
Nov 02 15:27:51.265582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.265553    5041 scope.go:122] "RemoveContainer" containerID="23706eb38ed8df0a9c02d35f98b40a3609f53bffbedd768575286f961dc2eb33"
Nov 02 15:27:51.336285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.336260    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.336485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.336453    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlhg\" (UniqueName: \"kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.336599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.336585    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.336646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.336608    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.447091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.442253    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlhg\" (UniqueName: \"kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.447091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.443602    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.447091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.443647    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.447091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.443798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.447091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.444856    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.450923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.450875    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.453496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.453471    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.465462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.465437    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlhg\" (UniqueName: \"kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg\") pod \"maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.516127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.516100    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:51.831531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:51.831500    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"]
Nov 02 15:27:51.832449 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:27:51.832418    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7759d79f_bc30_4b32_9c66_a96be695caa4.slice/crio-7dfc8fa6206694445ed5e876554630880237ddcc2928c12a382f8bede07f5784 WatchSource:0}: Error finding container 7dfc8fa6206694445ed5e876554630880237ddcc2928c12a382f8bede07f5784: Status 404 returned error can't find the container with id 7dfc8fa6206694445ed5e876554630880237ddcc2928c12a382f8bede07f5784
Nov 02 15:27:52.465457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:52.465417    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw" event={"ID":"7759d79f-bc30-4b32-9c66-a96be695caa4","Type":"ContainerStarted","Data":"2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef"}
Nov 02 15:27:52.465841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:52.465485    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:27:52.465841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:52.465506    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw" event={"ID":"7759d79f-bc30-4b32-9c66-a96be695caa4","Type":"ContainerStarted","Data":"7dfc8fa6206694445ed5e876554630880237ddcc2928c12a382f8bede07f5784"}
Nov 02 15:27:52.514119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:52.514068    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw" podStartSLOduration=1.221010345 podStartE2EDuration="1.514055761s" totalImagesPullingTime="293.045416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:52.510201249 +0000 UTC m=+1924.768270699" watchObservedRunningTime="2026-11-02 15:27:52.514055761 +0000 UTC m=+1924.772125211"
Nov 02 15:27:58.478852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:27:58.478820    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:28:06.770305 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:28:06.770264    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:28:36.770271 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:28:36.770236    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:28:58.139744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:28:58.139703    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"]
Nov 02 15:28:58.140287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:28:58.139918    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy" containerID="cri-o://924f51f5290810cf1799b09193959eddb5e6b1f7b05860938de64223f01e7b3c" gracePeriod=30
Nov 02 15:29:03.352031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.351997    5041 generic.go:356] "Generic (PLEG): container finished" podID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerID="924f51f5290810cf1799b09193959eddb5e6b1f7b05860938de64223f01e7b3c" exitCode=0
Nov 02 15:29:03.352514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.352217    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" event={"ID":"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe","Type":"ContainerDied","Data":"924f51f5290810cf1799b09193959eddb5e6b1f7b05860938de64223f01e7b3c"}
Nov 02 15:29:03.453612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.453579    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:29:03.595836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.595800    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.595933    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.595989    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596026    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596156    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596216    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596306    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596341    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:03.596414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596375    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:03.596528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596418    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596452    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token\") pod \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\" (UID: \"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe\") "
Nov 02 15:29:03.596634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596510    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:03.596688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596643    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:03.596870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.596569    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:03.597345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.597317    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.597482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.597351    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.597482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.597385    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.597482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.597400    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.597482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.597415    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.598480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.598454    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "kube-api-access-fg2wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:03.598613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.598594    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:03.598729 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.598706    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:03.598835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.598802    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token" pod "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" (UID: "f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:03.699322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.699240    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2wk\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-kube-api-access-fg2wk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.699322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.699271    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.699322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.699281    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:03.699322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:03.699289    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:04.363239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:04.363204    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp" event={"ID":"f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe","Type":"ContainerDied","Data":"a24d29319b578f376e67a433645923255812ad7055922aadd67fad3acc51638b"}
Nov 02 15:29:04.363722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:04.363266    5041 scope.go:122] "RemoveContainer" containerID="924f51f5290810cf1799b09193959eddb5e6b1f7b05860938de64223f01e7b3c"
Nov 02 15:29:04.363722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:04.363212    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"
Nov 02 15:29:04.430156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:04.430124    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"]
Nov 02 15:29:04.441535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:04.441508    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7xgqp"]
Nov 02 15:29:05.892031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:05.891998    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" path="/var/lib/kubelet/pods/f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe/volumes"
Nov 02 15:29:06.770417 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:29:06.770351    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:29:30.831982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:30.831734    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"]
Nov 02 15:29:30.833033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:30.832999    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4" podUID="fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" containerName="maas-api" containerID="cri-o://bb8bd8d697fbcd15f688bdf2b9dc9af84581712b9a1a1e8b60a009b33e576d88" gracePeriod=30
Nov 02 15:29:31.785341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.785016    5041 generic.go:356] "Generic (PLEG): container finished" podID="fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" containerID="bb8bd8d697fbcd15f688bdf2b9dc9af84581712b9a1a1e8b60a009b33e576d88" exitCode=0
Nov 02 15:29:31.785341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.785224    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4" event={"ID":"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee","Type":"ContainerDied","Data":"bb8bd8d697fbcd15f688bdf2b9dc9af84581712b9a1a1e8b60a009b33e576d88"}
Nov 02 15:29:31.810270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.810244    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:29:31.835828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.835790    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca\") pod \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") "
Nov 02 15:29:31.836245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.835973    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls\") pod \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") "
Nov 02 15:29:31.836245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.836122    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs\") pod \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") "
Nov 02 15:29:31.836352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.836236    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca" pod "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" (UID: "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:31.836828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.836792    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:31.838253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.838222    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs" pod "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" (UID: "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:31.838447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.838334    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls" pod "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" (UID: "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:31.937584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.937550    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58\" (UniqueName: \"kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58\") pod \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\" (UID: \"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee\") "
Nov 02 15:29:31.938287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.938261    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:31.938597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.938571    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:31.939929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:31.939905    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58" pod "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" (UID: "fcea4fe7-a375-4ffb-8a2d-8964ca4795ee"). InnerVolumeSpecName "kube-api-access-lwd58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:32.039965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:32.039895    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwd58\" (UniqueName: \"kubernetes.io/projected/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee-kube-api-access-lwd58\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:29:32.796532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:32.796502    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"
Nov 02 15:29:32.796702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:32.796534    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4" event={"ID":"fcea4fe7-a375-4ffb-8a2d-8964ca4795ee","Type":"ContainerDied","Data":"3d0cd34886baf85696d46c8898d189378b8ae6562216454dfe54b940386d38eb"}
Nov 02 15:29:32.796702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:32.796602    5041 scope.go:122] "RemoveContainer" containerID="bb8bd8d697fbcd15f688bdf2b9dc9af84581712b9a1a1e8b60a009b33e576d88"
Nov 02 15:29:32.852024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:32.852000    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"]
Nov 02 15:29:32.862886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:32.862859    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2bbcb5-7896b6b557-c7pc4"]
Nov 02 15:29:33.889554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:33.889521    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" path="/var/lib/kubelet/pods/fcea4fe7-a375-4ffb-8a2d-8964ca4795ee/volumes"
Nov 02 15:29:36.770664 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:29:36.770625    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.285196    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"]
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.286649    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" containerName="maas-api"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.286672    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" containerName="maas-api"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.286703    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.286711    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.286993    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcea4fe7-a375-4ffb-8a2d-8964ca4795ee" containerName="maas-api"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.287012    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8066585-1ef2-4e74-bc7b-d90e4b3c8cbe" containerName="istio-proxy"
Nov 02 15:29:46.299384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.292414    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.323797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.323448    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b955c5-dockercfg-7c5xw\""
Nov 02 15:29:46.323797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.323698    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b955c5\""
Nov 02 15:29:46.376444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.375901    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.376444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.376010    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkgt\" (UniqueName: \"kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt\") pod \"payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.408281 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.408212    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"]
Nov 02 15:29:46.477421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.477386    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.477613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.477476    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkgt\" (UniqueName: \"kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt\") pod \"payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.478119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.478099    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.510980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.510953    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkgt\" (UniqueName: \"kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt\") pod \"payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.643285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.643218    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:46.955075 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:29:46.955045    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09975717_b3b1_46ed_82b1_be1c2ca8403e.slice/crio-cdbd7c713c53be208cee6f969086c790e57fe11268458aeba6c82eb108bb6bfe WatchSource:0}: Error finding container cdbd7c713c53be208cee6f969086c790e57fe11268458aeba6c82eb108bb6bfe: Status 404 returned error can't find the container with id cdbd7c713c53be208cee6f969086c790e57fe11268458aeba6c82eb108bb6bfe
Nov 02 15:29:46.957127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.957099    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"]
Nov 02 15:29:46.968169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:46.968133    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" event={"ID":"09975717-b3b1-46ed-82b1-be1c2ca8403e","Type":"ContainerStarted","Data":"cdbd7c713c53be208cee6f969086c790e57fe11268458aeba6c82eb108bb6bfe"}
Nov 02 15:29:47.981569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:47.981532    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" event={"ID":"09975717-b3b1-46ed-82b1-be1c2ca8403e","Type":"ContainerStarted","Data":"1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a"}
Nov 02 15:29:47.982057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:47.981609    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:29:48.024683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:48.024630    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" podStartSLOduration=2.024614274 podStartE2EDuration="2.024614274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:48.021097027 +0000 UTC m=+2040.279166476" watchObservedRunningTime="2026-11-02 15:29:48.024614274 +0000 UTC m=+2040.282683750"
Nov 02 15:29:58.994140 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:29:58.994105    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:30:00.357843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.357800    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"]
Nov 02 15:30:00.365307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.365281    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.397590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.397537    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp5m\" (UniqueName: \"kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m\") pod \"maas-api-key-cleanup-29893890-lks79\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.397810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.397790    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-lks79\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.399128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.399106    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"]
Nov 02 15:30:00.405980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.405952    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:30:00.408508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.408485    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"]
Nov 02 15:30:00.408664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.408595    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.411406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.410992    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"]
Nov 02 15:30:00.418730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.416763    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.434618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.433408    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"]
Nov 02 15:30:00.444456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.442408    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"]
Nov 02 15:30:00.448405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.448378    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.463043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.461132    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"]
Nov 02 15:30:00.468262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.468224    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"]
Nov 02 15:30:00.499246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbq6g\" (UniqueName: \"kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g\") pod \"maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.499246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499244    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgrt\" (UniqueName: \"kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt\") pod \"maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.499472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499268    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.499600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499572    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp5m\" (UniqueName: \"kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m\") pod \"maas-api-key-cleanup-29893890-lks79\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.499676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499626    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-lks79\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.499747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499718    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vt4x\" (UniqueName: \"kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x\") pod \"maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.499808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499769    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.499902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.499882    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.500337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.500312    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-lks79\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.523530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.523501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp5m\" (UniqueName: \"kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m\") pod \"maas-api-key-cleanup-29893890-lks79\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.600932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.600908    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbq6g\" (UniqueName: \"kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g\") pod \"maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.601084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.600938    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgrt\" (UniqueName: \"kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt\") pod \"maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.601084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.600966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.601201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.601135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vt4x\" (UniqueName: \"kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x\") pod \"maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.601201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.601164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.601297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.601210    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.601654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.601623    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.601845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.601827    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.601888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.601833    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.624480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.624429    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vt4x\" (UniqueName: \"kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x\") pod \"maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:00.624714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.624692    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgrt\" (UniqueName: \"kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt\") pod \"maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.624758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.624703    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbq6g\" (UniqueName: \"kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g\") pod \"maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.700612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.700587    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:00.741411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.741378    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:00.751688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.751653    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:00.778434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:00.778411    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:01.018812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:01.018781    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"]
Nov 02 15:30:01.099586 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:30:01.099551    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461d7a45_fdb9_4b2c_b84a_681197534441.slice/crio-481c15f3077de1846e3290804e6c6c1c579f9283a11e0b9121a25cdffa08bc01 WatchSource:0}: Error finding container 481c15f3077de1846e3290804e6c6c1c579f9283a11e0b9121a25cdffa08bc01: Status 404 returned error can't find the container with id 481c15f3077de1846e3290804e6c6c1c579f9283a11e0b9121a25cdffa08bc01
Nov 02 15:30:01.100831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:01.100786    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"]
Nov 02 15:30:01.130995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:01.130947    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"]
Nov 02 15:30:01.142175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:01.142148    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerStarted","Data":"481c15f3077de1846e3290804e6c6c1c579f9283a11e0b9121a25cdffa08bc01"}
Nov 02 15:30:01.143546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:01.143521    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" event={"ID":"67470238-3ea9-49ac-9497-a67c6f1e9b09","Type":"ContainerStarted","Data":"30cbcc8a888da01b3ec3ae386fc1f4353967151deff2c6929eea3807d179f061"}
Nov 02 15:30:01.178527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:01.178506    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"]
Nov 02 15:30:02.154038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.154003    5041 generic.go:356] "Generic (PLEG): container finished" podID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerID="6aa27b95fb7a582aa9ab54c260e2bb1de4e6bd63f6c01fa2832355ca99e5a668" exitCode=60
Nov 02 15:30:02.154497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.154081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" event={"ID":"67470238-3ea9-49ac-9497-a67c6f1e9b09","Type":"ContainerDied","Data":"6aa27b95fb7a582aa9ab54c260e2bb1de4e6bd63f6c01fa2832355ca99e5a668"}
Nov 02 15:30:02.154497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.154306    5041 scope.go:122] "RemoveContainer" containerID="6aa27b95fb7a582aa9ab54c260e2bb1de4e6bd63f6c01fa2832355ca99e5a668"
Nov 02 15:30:02.156336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.156308    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerStarted","Data":"eb428a29cd214d16d85867ed89b22827c1a20c24d1dcac5ee4f243598effdf41"}
Nov 02 15:30:02.156484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.156356    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerStarted","Data":"c6e4a7b97771173cbf9efa386c0ce466cf57cf0f191dacfe509e7eda3de77714"}
Nov 02 15:30:02.158166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.158089    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerStarted","Data":"aecd9c912e37710c2be4edfa72e12198a462aa2940852538bca55cbcf96ab6b5"}
Nov 02 15:30:02.159826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.159800    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerStarted","Data":"372600cb404a961aeb2857892f5a3a8483f5399e817eda33a123d255f63ce053"}
Nov 02 15:30:02.159926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.159846    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerStarted","Data":"e06259e9c3d95756a4de8096d0be87e0b241d99633b6cd601874e0247c9db092"}
Nov 02 15:30:02.235538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.235490    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" podStartSLOduration=2.235477281 podStartE2EDuration="2.235477281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.231193144 +0000 UTC m=+2054.489262594" watchObservedRunningTime="2026-11-02 15:30:02.235477281 +0000 UTC m=+2054.493546732"
Nov 02 15:30:02.274694 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.274618    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" podStartSLOduration=2.274603514 podStartE2EDuration="2.274603514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.269775255 +0000 UTC m=+2054.527844705" watchObservedRunningTime="2026-11-02 15:30:02.274603514 +0000 UTC m=+2054.532672964"
Nov 02 15:30:02.310587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:02.310540    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" podStartSLOduration=2.310526119 podStartE2EDuration="2.310526119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.308214895 +0000 UTC m=+2054.566284345" watchObservedRunningTime="2026-11-02 15:30:02.310526119 +0000 UTC m=+2054.568595570"
Nov 02 15:30:03.170223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.170190    5041 generic.go:356] "Generic (PLEG): container finished" podID="321114da-7b16-4494-b58f-b408afc2af1c" containerID="372600cb404a961aeb2857892f5a3a8483f5399e817eda33a123d255f63ce053" exitCode=60
Nov 02 15:30:03.170672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.170229    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerDied","Data":"372600cb404a961aeb2857892f5a3a8483f5399e817eda33a123d255f63ce053"}
Nov 02 15:30:03.170672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.170549    5041 scope.go:122] "RemoveContainer" containerID="372600cb404a961aeb2857892f5a3a8483f5399e817eda33a123d255f63ce053"
Nov 02 15:30:03.172682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.172606    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" event={"ID":"67470238-3ea9-49ac-9497-a67c6f1e9b09","Type":"ContainerStarted","Data":"74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5"}
Nov 02 15:30:03.174791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.174429    5041 generic.go:356] "Generic (PLEG): container finished" podID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerID="eb428a29cd214d16d85867ed89b22827c1a20c24d1dcac5ee4f243598effdf41" exitCode=60
Nov 02 15:30:03.174791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.174634    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerDied","Data":"eb428a29cd214d16d85867ed89b22827c1a20c24d1dcac5ee4f243598effdf41"}
Nov 02 15:30:03.175009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.174867    5041 scope.go:122] "RemoveContainer" containerID="eb428a29cd214d16d85867ed89b22827c1a20c24d1dcac5ee4f243598effdf41"
Nov 02 15:30:03.176767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.176739    5041 generic.go:356] "Generic (PLEG): container finished" podID="461d7a45-fdb9-4b2c-b84a-681197534441" containerID="aecd9c912e37710c2be4edfa72e12198a462aa2940852538bca55cbcf96ab6b5" exitCode=60
Nov 02 15:30:03.176898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.176783    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerDied","Data":"aecd9c912e37710c2be4edfa72e12198a462aa2940852538bca55cbcf96ab6b5"}
Nov 02 15:30:03.177118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.177097    5041 scope.go:122] "RemoveContainer" containerID="aecd9c912e37710c2be4edfa72e12198a462aa2940852538bca55cbcf96ab6b5"
Nov 02 15:30:03.296085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:03.296042    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" podStartSLOduration=3.296030174 podStartE2EDuration="3.296030174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.292491772 +0000 UTC m=+2055.550561222" watchObservedRunningTime="2026-11-02 15:30:03.296030174 +0000 UTC m=+2055.554099623"
Nov 02 15:30:04.188372 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.188323    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerStarted","Data":"17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c"}
Nov 02 15:30:04.191186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.191156    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerStarted","Data":"741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0"}
Nov 02 15:30:04.193331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.193309    5041 generic.go:356] "Generic (PLEG): container finished" podID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerID="74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5" exitCode=60
Nov 02 15:30:04.193450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.193414    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" event={"ID":"67470238-3ea9-49ac-9497-a67c6f1e9b09","Type":"ContainerDied","Data":"74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5"}
Nov 02 15:30:04.193515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.193462    5041 scope.go:122] "RemoveContainer" containerID="6aa27b95fb7a582aa9ab54c260e2bb1de4e6bd63f6c01fa2832355ca99e5a668"
Nov 02 15:30:04.193669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.193651    5041 scope.go:122] "RemoveContainer" containerID="74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5"
Nov 02 15:30:04.193963 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:04.193937    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-lks79_odh-ai-gateway-infra(67470238-3ea9-49ac-9497-a67c6f1e9b09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09"
Nov 02 15:30:04.195598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:04.195581    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerStarted","Data":"c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec"}
Nov 02 15:30:05.207743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.207698    5041 generic.go:356] "Generic (PLEG): container finished" podID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerID="c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec" exitCode=60
Nov 02 15:30:05.208228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.207780    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerDied","Data":"c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec"}
Nov 02 15:30:05.208228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.207843    5041 scope.go:122] "RemoveContainer" containerID="eb428a29cd214d16d85867ed89b22827c1a20c24d1dcac5ee4f243598effdf41"
Nov 02 15:30:05.208228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.208129    5041 scope.go:122] "RemoveContainer" containerID="c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec"
Nov 02 15:30:05.208496 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:05.208473    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw_odh-ai-gateway-infra(ee4c4d33-a704-46ef-a90e-80b56b68df6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a"
Nov 02 15:30:05.210125 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.210098    5041 generic.go:356] "Generic (PLEG): container finished" podID="461d7a45-fdb9-4b2c-b84a-681197534441" containerID="17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c" exitCode=60
Nov 02 15:30:05.210216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.210181    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerDied","Data":"17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c"}
Nov 02 15:30:05.210468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.210449    5041 scope.go:122] "RemoveContainer" containerID="17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c"
Nov 02 15:30:05.210813 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:05.210793    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b_odh-ai-gateway-infra(461d7a45-fdb9-4b2c-b84a-681197534441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" podUID="461d7a45-fdb9-4b2c-b84a-681197534441"
Nov 02 15:30:05.212149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.212126    5041 generic.go:356] "Generic (PLEG): container finished" podID="321114da-7b16-4494-b58f-b408afc2af1c" containerID="741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0" exitCode=60
Nov 02 15:30:05.212262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.212206    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerDied","Data":"741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0"}
Nov 02 15:30:05.212459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.212438    5041 scope.go:122] "RemoveContainer" containerID="741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0"
Nov 02 15:30:05.212687 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:05.212665    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt_odh-ai-gateway-infra(321114da-7b16-4494-b58f-b408afc2af1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" podUID="321114da-7b16-4494-b58f-b408afc2af1c"
Nov 02 15:30:05.214243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.214226    5041 scope.go:122] "RemoveContainer" containerID="74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5"
Nov 02 15:30:05.214480 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:05.214458    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-lks79_odh-ai-gateway-infra(67470238-3ea9-49ac-9497-a67c6f1e9b09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09"
Nov 02 15:30:05.222396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.222359    5041 scope.go:122] "RemoveContainer" containerID="aecd9c912e37710c2be4edfa72e12198a462aa2940852538bca55cbcf96ab6b5"
Nov 02 15:30:05.239181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:05.239162    5041 scope.go:122] "RemoveContainer" containerID="372600cb404a961aeb2857892f5a3a8483f5399e817eda33a123d255f63ce053"
Nov 02 15:30:06.225344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:06.225312    5041 scope.go:122] "RemoveContainer" containerID="741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0"
Nov 02 15:30:06.226090 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:06.226060    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt_odh-ai-gateway-infra(321114da-7b16-4494-b58f-b408afc2af1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" podUID="321114da-7b16-4494-b58f-b408afc2af1c"
Nov 02 15:30:06.227526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:06.227504    5041 scope.go:122] "RemoveContainer" containerID="c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec"
Nov 02 15:30:06.227794 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:06.227769    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw_odh-ai-gateway-infra(ee4c4d33-a704-46ef-a90e-80b56b68df6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a"
Nov 02 15:30:06.229130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:06.229112    5041 scope.go:122] "RemoveContainer" containerID="17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c"
Nov 02 15:30:06.229341 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:06.229324    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b_odh-ai-gateway-infra(461d7a45-fdb9-4b2c-b84a-681197534441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" podUID="461d7a45-fdb9-4b2c-b84a-681197534441"
Nov 02 15:30:06.771685 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:06.771641    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:30:13.880709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:13.880673    5041 scope.go:122] "RemoveContainer" containerID="74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5"
Nov 02 15:30:15.314412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:15.314310    5041 generic.go:356] "Generic (PLEG): container finished" podID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerID="cdcd37fede212e41efb3205ab24f541e80d4ba34d5c21af1a36975c566285046" exitCode=60
Nov 02 15:30:15.314412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:15.314390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" event={"ID":"67470238-3ea9-49ac-9497-a67c6f1e9b09","Type":"ContainerDied","Data":"cdcd37fede212e41efb3205ab24f541e80d4ba34d5c21af1a36975c566285046"}
Nov 02 15:30:15.314900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:15.314448    5041 scope.go:122] "RemoveContainer" containerID="74fdac370788a5d409c618aeb4ee41f3e6b778765a3b1f0c3ebb3e7818bc98c5"
Nov 02 15:30:15.314900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:15.314871    5041 scope.go:122] "RemoveContainer" containerID="cdcd37fede212e41efb3205ab24f541e80d4ba34d5c21af1a36975c566285046"
Nov 02 15:30:15.315274 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:15.315254    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-lks79_odh-ai-gateway-infra(67470238-3ea9-49ac-9497-a67c6f1e9b09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09"
Nov 02 15:30:16.372204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.372173    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"]
Nov 02 15:30:16.551462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.551432    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:16.564314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.564283    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m\" (UniqueName: \"kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m\") pod \"67470238-3ea9-49ac-9497-a67c6f1e9b09\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") "
Nov 02 15:30:16.564485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.564355    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca\") pod \"67470238-3ea9-49ac-9497-a67c6f1e9b09\" (UID: \"67470238-3ea9-49ac-9497-a67c6f1e9b09\") "
Nov 02 15:30:16.564690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.564667    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca" pod "67470238-3ea9-49ac-9497-a67c6f1e9b09" (UID: "67470238-3ea9-49ac-9497-a67c6f1e9b09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:16.564976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.564955    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67470238-3ea9-49ac-9497-a67c6f1e9b09-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:16.566476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.566446    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m" pod "67470238-3ea9-49ac-9497-a67c6f1e9b09" (UID: "67470238-3ea9-49ac-9497-a67c6f1e9b09"). InnerVolumeSpecName "kube-api-access-pmp5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:16.666343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.666274    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp5m\" (UniqueName: \"kubernetes.io/projected/67470238-3ea9-49ac-9497-a67c6f1e9b09-kube-api-access-pmp5m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:16.879868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:16.879839    5041 scope.go:122] "RemoveContainer" containerID="c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec"
Nov 02 15:30:17.337761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.337739    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"
Nov 02 15:30:17.337876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.337768    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79" event={"ID":"67470238-3ea9-49ac-9497-a67c6f1e9b09","Type":"ContainerDied","Data":"30cbcc8a888da01b3ec3ae386fc1f4353967151deff2c6929eea3807d179f061"}
Nov 02 15:30:17.337876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.337851    5041 scope.go:122] "RemoveContainer" containerID="cdcd37fede212e41efb3205ab24f541e80d4ba34d5c21af1a36975c566285046"
Nov 02 15:30:17.340196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.340169    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerStarted","Data":"3d3f4504ffc3a8c42539b24cff37f21b07bf1b82134d51621ec39f2b252c1715"}
Nov 02 15:30:17.416103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.416073    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"]
Nov 02 15:30:17.444950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.444922    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-lks79"]
Nov 02 15:30:17.922764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.922730    5041 scope.go:122] "RemoveContainer" containerID="17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c"
Nov 02 15:30:17.927995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:17.927968    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" path="/var/lib/kubelet/pods/67470238-3ea9-49ac-9497-a67c6f1e9b09/volumes"
Nov 02 15:30:18.352591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:18.352561    5041 generic.go:356] "Generic (PLEG): container finished" podID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerID="3d3f4504ffc3a8c42539b24cff37f21b07bf1b82134d51621ec39f2b252c1715" exitCode=60
Nov 02 15:30:18.352775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:18.352647    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerDied","Data":"3d3f4504ffc3a8c42539b24cff37f21b07bf1b82134d51621ec39f2b252c1715"}
Nov 02 15:30:18.352775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:18.352725    5041 scope.go:122] "RemoveContainer" containerID="c206ae52683c1b2037ce73f0678b7a0bc5f83e3f4d09de18e1e5d47db15ba7ec"
Nov 02 15:30:18.353097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:18.353077    5041 scope.go:122] "RemoveContainer" containerID="3d3f4504ffc3a8c42539b24cff37f21b07bf1b82134d51621ec39f2b252c1715"
Nov 02 15:30:18.353501 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:18.353473    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw_odh-ai-gateway-infra(ee4c4d33-a704-46ef-a90e-80b56b68df6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a"
Nov 02 15:30:18.519821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:18.519724    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"]
Nov 02 15:30:18.880285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:18.880214    5041 scope.go:122] "RemoveContainer" containerID="741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0"
Nov 02 15:30:19.366269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.366239    5041 generic.go:356] "Generic (PLEG): container finished" podID="461d7a45-fdb9-4b2c-b84a-681197534441" containerID="82f61118425e0eba6902373826c57de0e9c79907fb1bac16c9efb461dce0e9ff" exitCode=60
Nov 02 15:30:19.366542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.366316    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerDied","Data":"82f61118425e0eba6902373826c57de0e9c79907fb1bac16c9efb461dce0e9ff"}
Nov 02 15:30:19.366542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.366429    5041 scope.go:122] "RemoveContainer" containerID="17940a3d0e7ac2e19effb475da7532e141dde4c21a3037f073bb931995f7c28c"
Nov 02 15:30:19.367632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.366856    5041 scope.go:122] "RemoveContainer" containerID="82f61118425e0eba6902373826c57de0e9c79907fb1bac16c9efb461dce0e9ff"
Nov 02 15:30:19.367632 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:19.367222    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b_odh-ai-gateway-infra(461d7a45-fdb9-4b2c-b84a-681197534441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" podUID="461d7a45-fdb9-4b2c-b84a-681197534441"
Nov 02 15:30:19.368876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.368853    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerStarted","Data":"1d37053da891b9ba5a7ba9d68961c29c7db3d33700bdd77b611c2e4ab49227e9"}
Nov 02 15:30:19.584692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.584665    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:19.671872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.671842    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca\") pod \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") "
Nov 02 15:30:19.672063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.671996    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt\" (UniqueName: \"kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt\") pod \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\" (UID: \"ee4c4d33-a704-46ef-a90e-80b56b68df6a\") "
Nov 02 15:30:19.672241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.672216    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca" pod "ee4c4d33-a704-46ef-a90e-80b56b68df6a" (UID: "ee4c4d33-a704-46ef-a90e-80b56b68df6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.672697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.672675    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4c4d33-a704-46ef-a90e-80b56b68df6a-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.674190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.674159    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt" pod "ee4c4d33-a704-46ef-a90e-80b56b68df6a" (UID: "ee4c4d33-a704-46ef-a90e-80b56b68df6a"). InnerVolumeSpecName "kube-api-access-bfgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.774298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:19.774263    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgrt\" (UniqueName: \"kubernetes.io/projected/ee4c4d33-a704-46ef-a90e-80b56b68df6a-kube-api-access-bfgrt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.381204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.381173    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"
Nov 02 15:30:20.381416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.381178    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw" event={"ID":"ee4c4d33-a704-46ef-a90e-80b56b68df6a","Type":"ContainerDied","Data":"c6e4a7b97771173cbf9efa386c0ce466cf57cf0f191dacfe509e7eda3de77714"}
Nov 02 15:30:20.381416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.381323    5041 scope.go:122] "RemoveContainer" containerID="3d3f4504ffc3a8c42539b24cff37f21b07bf1b82134d51621ec39f2b252c1715"
Nov 02 15:30:20.396209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.396181    5041 generic.go:356] "Generic (PLEG): container finished" podID="321114da-7b16-4494-b58f-b408afc2af1c" containerID="1d37053da891b9ba5a7ba9d68961c29c7db3d33700bdd77b611c2e4ab49227e9" exitCode=60
Nov 02 15:30:20.396441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.396346    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerDied","Data":"1d37053da891b9ba5a7ba9d68961c29c7db3d33700bdd77b611c2e4ab49227e9"}
Nov 02 15:30:20.396885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.396861    5041 scope.go:122] "RemoveContainer" containerID="1d37053da891b9ba5a7ba9d68961c29c7db3d33700bdd77b611c2e4ab49227e9"
Nov 02 15:30:20.397205 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:20.397172    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt_odh-ai-gateway-infra(321114da-7b16-4494-b58f-b408afc2af1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" podUID="321114da-7b16-4494-b58f-b408afc2af1c"
Nov 02 15:30:20.400323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.400296    5041 scope.go:122] "RemoveContainer" containerID="741cf232403342bc221bd0fa30c1b7eee03cdecf62acbf041925745bec2ebcc0"
Nov 02 15:30:20.433402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.433359    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"]
Nov 02 15:30:20.482043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.482006    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"]
Nov 02 15:30:20.487351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.487323    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"]
Nov 02 15:30:20.508187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.508150    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-69ff88-29893890-tkzjw"]
Nov 02 15:30:20.629559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.629533    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:20.801956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.801926    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g\" (UniqueName: \"kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g\") pod \"461d7a45-fdb9-4b2c-b84a-681197534441\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") "
Nov 02 15:30:20.802145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.802060    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca\") pod \"461d7a45-fdb9-4b2c-b84a-681197534441\" (UID: \"461d7a45-fdb9-4b2c-b84a-681197534441\") "
Nov 02 15:30:20.802525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.802492    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca" pod "461d7a45-fdb9-4b2c-b84a-681197534441" (UID: "461d7a45-fdb9-4b2c-b84a-681197534441"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.802865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.802842    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/461d7a45-fdb9-4b2c-b84a-681197534441-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.804431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.804405    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g" pod "461d7a45-fdb9-4b2c-b84a-681197534441" (UID: "461d7a45-fdb9-4b2c-b84a-681197534441"). InnerVolumeSpecName "kube-api-access-fbq6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.903615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:20.903586    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbq6g\" (UniqueName: \"kubernetes.io/projected/461d7a45-fdb9-4b2c-b84a-681197534441-kube-api-access-fbq6g\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.408377 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.408330    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"
Nov 02 15:30:21.408563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.408390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b" event={"ID":"461d7a45-fdb9-4b2c-b84a-681197534441","Type":"ContainerDied","Data":"481c15f3077de1846e3290804e6c6c1c579f9283a11e0b9121a25cdffa08bc01"}
Nov 02 15:30:21.408563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.408451    5041 scope.go:122] "RemoveContainer" containerID="82f61118425e0eba6902373826c57de0e9c79907fb1bac16c9efb461dce0e9ff"
Nov 02 15:30:21.525051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.523084    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"]
Nov 02 15:30:21.594441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.592888    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-31c816-29893890-89n7b"]
Nov 02 15:30:21.663863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.663796    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:21.816710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.816677    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca\") pod \"321114da-7b16-4494-b58f-b408afc2af1c\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") "
Nov 02 15:30:21.816960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.816938    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x\" (UniqueName: \"kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x\") pod \"321114da-7b16-4494-b58f-b408afc2af1c\" (UID: \"321114da-7b16-4494-b58f-b408afc2af1c\") "
Nov 02 15:30:21.817029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.817010    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca" pod "321114da-7b16-4494-b58f-b408afc2af1c" (UID: "321114da-7b16-4494-b58f-b408afc2af1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.817666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.817642    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321114da-7b16-4494-b58f-b408afc2af1c-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.819239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.819203    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x" pod "321114da-7b16-4494-b58f-b408afc2af1c" (UID: "321114da-7b16-4494-b58f-b408afc2af1c"). InnerVolumeSpecName "kube-api-access-5vt4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.903278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.903247    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" path="/var/lib/kubelet/pods/461d7a45-fdb9-4b2c-b84a-681197534441/volumes"
Nov 02 15:30:21.903750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.903735    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" path="/var/lib/kubelet/pods/ee4c4d33-a704-46ef-a90e-80b56b68df6a/volumes"
Nov 02 15:30:21.918622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:21.918599    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vt4x\" (UniqueName: \"kubernetes.io/projected/321114da-7b16-4494-b58f-b408afc2af1c-kube-api-access-5vt4x\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.420739 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:22.420716    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"
Nov 02 15:30:22.420739 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:22.420733    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt" event={"ID":"321114da-7b16-4494-b58f-b408afc2af1c","Type":"ContainerDied","Data":"e06259e9c3d95756a4de8096d0be87e0b241d99633b6cd601874e0247c9db092"}
Nov 02 15:30:22.420946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:22.420784    5041 scope.go:122] "RemoveContainer" containerID="1d37053da891b9ba5a7ba9d68961c29c7db3d33700bdd77b611c2e4ab49227e9"
Nov 02 15:30:22.474585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:22.474559    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"]
Nov 02 15:30:22.485698 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:22.485675    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cfc63a-29893890-9btqt"]
Nov 02 15:30:23.891482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:23.891446    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321114da-7b16-4494-b58f-b408afc2af1c" path="/var/lib/kubelet/pods/321114da-7b16-4494-b58f-b408afc2af1c/volumes"
Nov 02 15:30:28.154012 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:28.153970    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"]
Nov 02 15:30:28.154487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:28.154159    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy" containerID="cri-o://aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c" gracePeriod=30
Nov 02 15:30:33.477924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.477893    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:30:33.480582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480553    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.480725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480624    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.480725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480657    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.480725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480701    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.480912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480771    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:33.480912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480787    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.480912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480854    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.480912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480890    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.481121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480911    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:33.481121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.480931    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.481121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.481006    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token\") pod \"df478ae8-7689-466f-8158-8edf00a0a2cb\" (UID: \"df478ae8-7689-466f-8158-8edf00a0a2cb\") "
Nov 02 15:30:33.481121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.481091    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:33.481337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.481178    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:33.481337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.481209    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:33.482051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.482026    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.482051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.482053    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.482256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.482077    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.482256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.482093    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df478ae8-7689-466f-8158-8edf00a0a2cb-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.482256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.482106    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.483135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.483109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:33.483257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.483169    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:33.483257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.483204    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:33.483354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.483247    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj" pod "df478ae8-7689-466f-8158-8edf00a0a2cb" (UID: "df478ae8-7689-466f-8158-8edf00a0a2cb"). InnerVolumeSpecName "kube-api-access-zdwqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:33.583429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.583398    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.583429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.583425    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdwqj\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-kube-api-access-zdwqj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.583429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.583435    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.583752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.583445    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df478ae8-7689-466f-8158-8edf00a0a2cb-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:30:33.678771 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.678739    5041 generic.go:356] "Generic (PLEG): container finished" podID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerID="aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c" exitCode=0
Nov 02 15:30:33.678950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.678784    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"
Nov 02 15:30:33.678950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.678812    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" event={"ID":"df478ae8-7689-466f-8158-8edf00a0a2cb","Type":"ContainerDied","Data":"aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c"}
Nov 02 15:30:33.678950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.678858    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh" event={"ID":"df478ae8-7689-466f-8158-8edf00a0a2cb","Type":"ContainerDied","Data":"da13df3c34de7f24f8c66d7b0e0adc9b150be4f99be01835a66c044de335d303"}
Nov 02 15:30:33.678950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.678882    5041 scope.go:122] "RemoveContainer" containerID="aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c"
Nov 02 15:30:33.691849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.691829    5041 scope.go:122] "RemoveContainer" containerID="aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c"
Nov 02 15:30:33.692144 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:33.692114    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c\": container with ID starting with aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c not found: ID does not exist" containerID="aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c"
Nov 02 15:30:33.692209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.692155    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c"} err="failed to get container status \"aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c\": rpc error: code = NotFound desc = could not find container \"aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c\": container with ID starting with aeefd6c6854dd52ea9ea10105fb6dbd34f2beeb34817ff288471fa8e02b9f94c not found: ID does not exist"
Nov 02 15:30:33.735198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.735176    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"]
Nov 02 15:30:33.740236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.740215    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-q2dsh"]
Nov 02 15:30:33.890996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:33.890964    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" path="/var/lib/kubelet/pods/df478ae8-7689-466f-8158-8edf00a0a2cb/volumes"
Nov 02 15:30:36.771415 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:30:36.771337    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:30:42.058767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.058733    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"]
Nov 02 15:30:42.059782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059758    5041 cpu_manager.go:414] "containerMap: removing container" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.059782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059779    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059801    5041 cpu_manager.go:414] "containerMap: removing container" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059806    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059813    5041 cpu_manager.go:414] "containerMap: removing container" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059818    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059827    5041 cpu_manager.go:414] "containerMap: removing container" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059833    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059849    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059854    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059871    5041 cpu_manager.go:414] "containerMap: removing container" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059878    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059890    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059895    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059927    5041 cpu_manager.go:414] "containerMap: removing container" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059934    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059949    5041 cpu_manager.go:414] "containerMap: removing container" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:30:42.060007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.059954    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060125    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060135    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060140    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060151    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060161    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060170    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060178    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060185    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060191    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="df478ae8-7689-466f-8158-8edf00a0a2cb" containerName="istio-proxy"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060197    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.060616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.060206    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:30:42.064560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.064540    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.092428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.092409    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2abeaa-openshift-default-dockercfg-smrsn\""
Nov 02 15:30:42.129153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.129129    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"]
Nov 02 15:30:42.133927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.133895    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134072    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134183    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5p4b\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134211    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134293    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134323    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134376    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134459    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.134600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.134492    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.235879    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.235942    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236025    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236152    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236264    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5p4b\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236294    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236399    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236431    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.236470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.237064    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.237619    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.238184    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.238524    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.240388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.239763    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.257941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.253942    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.263339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.263238    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.266101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.266070    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5p4b\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.266242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.266114    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.381566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.381486    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:42.712116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.712085    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"]
Nov 02 15:30:42.712513 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:30:42.712494    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46212a2_f445_4920_999b_cb38375d9927.slice/crio-663cb90f26d0b719f65a5e6e21ae77ce6c5e4eb3123365ab10ee782dee739900 WatchSource:0}: Error finding container 663cb90f26d0b719f65a5e6e21ae77ce6c5e4eb3123365ab10ee782dee739900: Status 404 returned error can't find the container with id 663cb90f26d0b719f65a5e6e21ae77ce6c5e4eb3123365ab10ee782dee739900
Nov 02 15:30:42.715050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.715011    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:42.715156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.715082    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:42.715156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.715111    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:42.810651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:42.810621    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" event={"ID":"e46212a2-f445-4920-999b-cb38375d9927","Type":"ContainerStarted","Data":"663cb90f26d0b719f65a5e6e21ae77ce6c5e4eb3123365ab10ee782dee739900"}
Nov 02 15:30:43.848303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:43.848261    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" event={"ID":"e46212a2-f445-4920-999b-cb38375d9927","Type":"ContainerStarted","Data":"476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d"}
Nov 02 15:30:43.975032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:43.974965    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podStartSLOduration=1.974948354 podStartE2EDuration="1.974948354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:43.906018164 +0000 UTC m=+2096.164087615" watchObservedRunningTime="2026-11-02 15:30:43.974948354 +0000 UTC m=+2096.233017803"
Nov 02 15:30:44.382152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:44.382109    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:44.383765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:44.383736    5041 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:30:44.383895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:44.383784    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 15:30:45.381978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:45.381942    5041 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:30:45.382468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:45.382002    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 15:30:46.382532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:46.382495    5041 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:30:46.383022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:46.382581    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 15:30:47.381778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:47.381745    5041 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:30:47.381977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:47.381795    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 15:30:48.382769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:48.382731    5041 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:30:48.383253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:48.382790    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 15:30:49.045715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:49.045601    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:30:49.073071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:49.048765    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:30:49.389943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:49.389851    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:49.390821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:49.390761    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:30:49.391728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:30:49.391700    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:31:06.770649 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:31:06.770575    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:31:33.428257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.428224    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"]
Nov 02 15:31:33.429808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429778    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:31:33.429808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429803    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4c4d33-a704-46ef-a90e-80b56b68df6a" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429835    5041 cpu_manager.go:414] "containerMap: removing container" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429840    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429854    5041 cpu_manager.go:414] "containerMap: removing container" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429859    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429886    5041 cpu_manager.go:414] "containerMap: removing container" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.429892    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321114da-7b16-4494-b58f-b408afc2af1c" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.430017    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="461d7a45-fdb9-4b2c-b84a-681197534441" containerName="cleanup"
Nov 02 15:31:33.430027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.430027    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="67470238-3ea9-49ac-9497-a67c6f1e9b09" containerName="cleanup"
Nov 02 15:31:33.435475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.435452    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.456595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.456569    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"]
Nov 02 15:31:33.462081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.462054    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-b55da1\""
Nov 02 15:31:33.462309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.462286    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-b55da1-dockercfg-cbfgb\""
Nov 02 15:31:33.594387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.591031    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45kx\" (UniqueName: \"kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx\") pod \"payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.594387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.591171    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.695190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.695098    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.695716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.695680    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45kx\" (UniqueName: \"kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx\") pod \"payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.696529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.696505    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.748259 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.748223    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45kx\" (UniqueName: \"kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx\") pod \"payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:33.753748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:33.753661    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:34.208785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:34.208716    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"]
Nov 02 15:31:34.373853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:34.373818    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" event={"ID":"1d0bf2a3-8dfb-494b-a765-3317b87c6b63","Type":"ContainerStarted","Data":"5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77"}
Nov 02 15:31:34.374001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:34.373873    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" event={"ID":"1d0bf2a3-8dfb-494b-a765-3317b87c6b63","Type":"ContainerStarted","Data":"2d805122f1f9e2d7d414d8402641d88635d26339290fbe0eed15d36936f0de91"}
Nov 02 15:31:34.374001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:34.373914    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:36.771027 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:31:36.770992    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:31:45.396812 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:45.396778    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:31:45.439984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:45.439937    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" podStartSLOduration=12.439923127 podStartE2EDuration="12.439923127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:34.423322179 +0000 UTC m=+2146.681391629" watchObservedRunningTime="2026-11-02 15:31:45.439923127 +0000 UTC m=+2157.697992601"
Nov 02 15:31:59.130153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:59.130118    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"]
Nov 02 15:31:59.130866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:59.130334    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b" podUID="c4af444f-93fa-4ea2-a652-fab7df58fbc2" containerName="maas-api" containerID="cri-o://cf1468cd0c4c01b8e013c44571480fdcf22581ad9b13e5f9d9b6725ebd0b2d94" gracePeriod=30
Nov 02 15:31:59.749776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:59.744890    5041 generic.go:356] "Generic (PLEG): container finished" podID="c4af444f-93fa-4ea2-a652-fab7df58fbc2" containerID="cf1468cd0c4c01b8e013c44571480fdcf22581ad9b13e5f9d9b6725ebd0b2d94" exitCode=0
Nov 02 15:31:59.749776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:59.745298    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b" event={"ID":"c4af444f-93fa-4ea2-a652-fab7df58fbc2","Type":"ContainerDied","Data":"cf1468cd0c4c01b8e013c44571480fdcf22581ad9b13e5f9d9b6725ebd0b2d94"}
Nov 02 15:31:59.914078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:31:59.914039    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:32:00.015427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.011304    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls\") pod \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") "
Nov 02 15:32:00.015427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.011443    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs\") pod \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") "
Nov 02 15:32:00.015427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.011629    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca\") pod \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") "
Nov 02 15:32:00.015427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.011841    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9\" (UniqueName: \"kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9\") pod \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\" (UID: \"c4af444f-93fa-4ea2-a652-fab7df58fbc2\") "
Nov 02 15:32:00.015427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.014066    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca" pod "c4af444f-93fa-4ea2-a652-fab7df58fbc2" (UID: "c4af444f-93fa-4ea2-a652-fab7df58fbc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:00.015792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.015706    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9" pod "c4af444f-93fa-4ea2-a652-fab7df58fbc2" (UID: "c4af444f-93fa-4ea2-a652-fab7df58fbc2"). InnerVolumeSpecName "kube-api-access-b5qb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:00.026688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.026656    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs" pod "c4af444f-93fa-4ea2-a652-fab7df58fbc2" (UID: "c4af444f-93fa-4ea2-a652-fab7df58fbc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:00.028376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.028345    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls" pod "c4af444f-93fa-4ea2-a652-fab7df58fbc2" (UID: "c4af444f-93fa-4ea2-a652-fab7df58fbc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:00.117544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.117501    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qb9\" (UniqueName: \"kubernetes.io/projected/c4af444f-93fa-4ea2-a652-fab7df58fbc2-kube-api-access-b5qb9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:00.117544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.117554    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:00.117807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.117575    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4af444f-93fa-4ea2-a652-fab7df58fbc2-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:00.117807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.117598    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4af444f-93fa-4ea2-a652-fab7df58fbc2-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:00.755811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.755782    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"
Nov 02 15:32:00.756257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.755821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b" event={"ID":"c4af444f-93fa-4ea2-a652-fab7df58fbc2","Type":"ContainerDied","Data":"02da6d3d8499b94b60df44a6594232cf7e321b26021d635ba1bb395a2c642d92"}
Nov 02 15:32:00.756257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.755883    5041 scope.go:122] "RemoveContainer" containerID="cf1468cd0c4c01b8e013c44571480fdcf22581ad9b13e5f9d9b6725ebd0b2d94"
Nov 02 15:32:00.814810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.814781    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"]
Nov 02 15:32:00.819506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:00.819480    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f975ed-649db7c47b-5p82b"]
Nov 02 15:32:01.890866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:01.890833    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4af444f-93fa-4ea2-a652-fab7df58fbc2" path="/var/lib/kubelet/pods/c4af444f-93fa-4ea2-a652-fab7df58fbc2/volumes"
Nov 02 15:32:03.782928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:03.782889    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"]
Nov 02 15:32:03.783315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:03.783076    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy" containerID="cri-o://477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a" gracePeriod=30
Nov 02 15:32:06.770174 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:06.770137    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:32:09.109299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.109274    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:32:09.248476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248444    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248585    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248675    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248719    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248754    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248796    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248888    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.248970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248927    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.248970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.248969    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.249143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249110    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs\") pod \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\" (UID: \"e05f53bc-3e54-4137-a7ab-ba0b6c06858a\") "
Nov 02 15:32:09.249275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249251    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.249422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.249504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249456    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:09.249563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249528    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.249897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249879    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.249897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249896    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.250043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249906    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.250043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249914    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.250043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.249922    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.251100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.251077    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:09.251232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.251214    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.251283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.251249    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:09.251283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.251255    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m" pod "e05f53bc-3e54-4137-a7ab-ba0b6c06858a" (UID: "e05f53bc-3e54-4137-a7ab-ba0b6c06858a"). InnerVolumeSpecName "kube-api-access-8sw8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:09.351217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.351135    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sw8m\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-kube-api-access-8sw8m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.351217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.351168    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.351217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.351181    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.351217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.351193    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e05f53bc-3e54-4137-a7ab-ba0b6c06858a-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.885568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.885536    5041 generic.go:356] "Generic (PLEG): container finished" podID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerID="477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a" exitCode=0
Nov 02 15:32:09.885759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.885639    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"
Nov 02 15:32:09.892439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.892407    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" event={"ID":"e05f53bc-3e54-4137-a7ab-ba0b6c06858a","Type":"ContainerDied","Data":"477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a"}
Nov 02 15:32:09.892573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.892464    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb" event={"ID":"e05f53bc-3e54-4137-a7ab-ba0b6c06858a","Type":"ContainerDied","Data":"bdfe0deed4ba209786417d1ec49af5c015d50ccae0e1138edbe2c5ecfe1cd057"}
Nov 02 15:32:09.892573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.892496    5041 scope.go:122] "RemoveContainer" containerID="477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a"
Nov 02 15:32:09.905520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.905502    5041 scope.go:122] "RemoveContainer" containerID="477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a"
Nov 02 15:32:09.905791 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:09.905768    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a\": container with ID starting with 477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a not found: ID does not exist" containerID="477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a"
Nov 02 15:32:09.905866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.905802    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a"} err="failed to get container status \"477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a\": rpc error: code = NotFound desc = could not find container \"477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a\": container with ID starting with 477ca4e5dead09b6b625666e78e812127fda3239777fddf8d411e649adb2754a not found: ID does not exist"
Nov 02 15:32:09.950926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.950903    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"]
Nov 02 15:32:09.961873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:09.961851    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-qp5cb"]
Nov 02 15:32:11.891004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:11.890968    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" path="/var/lib/kubelet/pods/e05f53bc-3e54-4137-a7ab-ba0b6c06858a/volumes"
Nov 02 15:32:18.563024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:18.562988    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"]
Nov 02 15:32:18.597732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:18.597701    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"]
Nov 02 15:32:23.563427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.563359    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk" podUID="cdb38061-bac4-4186-9588-0e297d26fb1e" containerName="payload-processing" containerID="cri-o://97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b" gracePeriod=25
Nov 02 15:32:23.598675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.598644    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" podUID="836a1569-607e-49e7-8af2-b9053eda759c" containerName="payload-pre-processing" containerID="cri-o://27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536" gracePeriod=25
Nov 02 15:32:23.868784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.868758    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:32:23.929968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.929932    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247\" (UniqueName: \"kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247\") pod \"cdb38061-bac4-4186-9588-0e297d26fb1e\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") "
Nov 02 15:32:23.930126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.930113    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume\") pod \"cdb38061-bac4-4186-9588-0e297d26fb1e\" (UID: \"cdb38061-bac4-4186-9588-0e297d26fb1e\") "
Nov 02 15:32:23.930463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.930437    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume" pod "cdb38061-bac4-4186-9588-0e297d26fb1e" (UID: "cdb38061-bac4-4186-9588-0e297d26fb1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:23.930613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.930595    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb38061-bac4-4186-9588-0e297d26fb1e-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:23.932025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:23.932002    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247" pod "cdb38061-bac4-4186-9588-0e297d26fb1e" (UID: "cdb38061-bac4-4186-9588-0e297d26fb1e"). InnerVolumeSpecName "kube-api-access-nm247". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:24.032040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.032011    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm247\" (UniqueName: \"kubernetes.io/projected/cdb38061-bac4-4186-9588-0e297d26fb1e-kube-api-access-nm247\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:24.062426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.062398    5041 generic.go:356] "Generic (PLEG): container finished" podID="cdb38061-bac4-4186-9588-0e297d26fb1e" containerID="97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b" exitCode=0
Nov 02 15:32:24.062532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.062443    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"
Nov 02 15:32:24.062532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.062485    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk" event={"ID":"cdb38061-bac4-4186-9588-0e297d26fb1e","Type":"ContainerDied","Data":"97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b"}
Nov 02 15:32:24.062628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.062542    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk" event={"ID":"cdb38061-bac4-4186-9588-0e297d26fb1e","Type":"ContainerDied","Data":"dfa2b02e26e124a70de5a25f7a68a877120eaeee1c0dd95dc6ba94b4b379663e"}
Nov 02 15:32:24.062628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.062570    5041 scope.go:122] "RemoveContainer" containerID="97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b"
Nov 02 15:32:24.074324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.074306    5041 scope.go:122] "RemoveContainer" containerID="97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b"
Nov 02 15:32:24.074622 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:24.074603    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b\": container with ID starting with 97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b not found: ID does not exist" containerID="97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b"
Nov 02 15:32:24.074670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.074628    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b"} err="failed to get container status \"97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b\": rpc error: code = NotFound desc = could not find container \"97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b\": container with ID starting with 97fe8c2953f5eaaaff7bf550a66c85276b19a30170735d6b67ba3cbdc717808b not found: ID does not exist"
Nov 02 15:32:24.117501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.117477    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"]
Nov 02 15:32:24.124630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.124609    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2abeaa-5fc47b87f5-rbsbk"]
Nov 02 15:32:24.238677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:24.238652    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" podUID="836a1569-607e-49e7-8af2-b9053eda759c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:32:25.891725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:25.891689    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb38061-bac4-4186-9588-0e297d26fb1e" path="/var/lib/kubelet/pods/cdb38061-bac4-4186-9588-0e297d26fb1e/volumes"
Nov 02 15:32:32.424399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.424287    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"]
Nov 02 15:32:32.424786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.424535    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" podUID="1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" containerName="maas-api" containerID="cri-o://f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d" gracePeriod=30
Nov 02 15:32:32.818617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.818577    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:32:32.940390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.934897    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca\") pod \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") "
Nov 02 15:32:32.940390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.935073    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\") pod \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") "
Nov 02 15:32:32.940390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.935226    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7\" (UniqueName: \"kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7\") pod \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") "
Nov 02 15:32:32.940390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.935307    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs\") pod \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\" (UID: \"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7\") "
Nov 02 15:32:32.942248 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.942054    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca" pod "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" (UID: "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:32.962737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.962694    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs" pod "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" (UID: "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:32.965586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.965350    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls" pod "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" (UID: "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:32.965586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:32.965414    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7" pod "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" (UID: "1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7"). InnerVolumeSpecName "kube-api-access-7mxt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:33.038003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.037911    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxt7\" (UniqueName: \"kubernetes.io/projected/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-kube-api-access-7mxt7\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:33.038003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.037947    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:33.038003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.037964    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:33.038003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.037978    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:33.150903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.150862    5041 generic.go:356] "Generic (PLEG): container finished" podID="1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" containerID="f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d" exitCode=0
Nov 02 15:32:33.151179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.151157    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" event={"ID":"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7","Type":"ContainerDied","Data":"f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d"}
Nov 02 15:32:33.151268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.151252    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8" event={"ID":"1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7","Type":"ContainerDied","Data":"7c869b7b6bed799ee8728d3ff5805b13eca6f14fbea3727dad73a9ff0d929eb6"}
Nov 02 15:32:33.151332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.151302    5041 scope.go:122] "RemoveContainer" containerID="f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d"
Nov 02 15:32:33.151551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.151532    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"
Nov 02 15:32:33.167513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.167487    5041 scope.go:122] "RemoveContainer" containerID="f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d"
Nov 02 15:32:33.167886 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:33.167857    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d\": container with ID starting with f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d not found: ID does not exist" containerID="f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d"
Nov 02 15:32:33.168001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.167891    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d"} err="failed to get container status \"f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d\": rpc error: code = NotFound desc = could not find container \"f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d\": container with ID starting with f8763951248fec3d8ec720b0f1494ef43933b1a1f67ff097311f2e8a59df905d not found: ID does not exist"
Nov 02 15:32:33.206293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.206269    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"]
Nov 02 15:32:33.218213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.218189    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a03e84-89fdc8577-4ddw8"]
Nov 02 15:32:33.890980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:33.890946    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" path="/var/lib/kubelet/pods/1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7/volumes"
Nov 02 15:32:34.237971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:34.237937    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" podUID="836a1569-607e-49e7-8af2-b9053eda759c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:32:36.770237 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:36.770198    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:32:43.909206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:43.909181    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:32:44.021550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.021515    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume\") pod \"836a1569-607e-49e7-8af2-b9053eda759c\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") "
Nov 02 15:32:44.021704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.021600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk\" (UniqueName: \"kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk\") pod \"836a1569-607e-49e7-8af2-b9053eda759c\" (UID: \"836a1569-607e-49e7-8af2-b9053eda759c\") "
Nov 02 15:32:44.021888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.021847    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume" pod "836a1569-607e-49e7-8af2-b9053eda759c" (UID: "836a1569-607e-49e7-8af2-b9053eda759c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:44.023676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.023651    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk" pod "836a1569-607e-49e7-8af2-b9053eda759c" (UID: "836a1569-607e-49e7-8af2-b9053eda759c"). InnerVolumeSpecName "kube-api-access-v8mjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:44.124269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.124212    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a1569-607e-49e7-8af2-b9053eda759c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:44.124269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.124246    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mjk\" (UniqueName: \"kubernetes.io/projected/836a1569-607e-49e7-8af2-b9053eda759c-kube-api-access-v8mjk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:44.257547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.257512    5041 generic.go:356] "Generic (PLEG): container finished" podID="836a1569-607e-49e7-8af2-b9053eda759c" containerID="27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536" exitCode=0
Nov 02 15:32:44.257717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.257563    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"
Nov 02 15:32:44.257717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.257595    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" event={"ID":"836a1569-607e-49e7-8af2-b9053eda759c","Type":"ContainerDied","Data":"27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536"}
Nov 02 15:32:44.257717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.257653    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp" event={"ID":"836a1569-607e-49e7-8af2-b9053eda759c","Type":"ContainerDied","Data":"4aeb4dda64974a0923753b79368902acf949c0a9ccff2d5bc9a79013dee66285"}
Nov 02 15:32:44.257717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.257677    5041 scope.go:122] "RemoveContainer" containerID="27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536"
Nov 02 15:32:44.269757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.269736    5041 scope.go:122] "RemoveContainer" containerID="27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536"
Nov 02 15:32:44.270046 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:44.270025    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536\": container with ID starting with 27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536 not found: ID does not exist" containerID="27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536"
Nov 02 15:32:44.270131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.270051    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536"} err="failed to get container status \"27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536\": rpc error: code = NotFound desc = could not find container \"27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536\": container with ID starting with 27402ce2638eef566956b6ad1de35da77598bcbdac6cabdb34b0d74698b38536 not found: ID does not exist"
Nov 02 15:32:44.310610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.310587    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"]
Nov 02 15:32:44.321776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:44.321754    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2abeaa-7564ccb7d7-p8pkp"]
Nov 02 15:32:45.890693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:45.890657    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836a1569-607e-49e7-8af2-b9053eda759c" path="/var/lib/kubelet/pods/836a1569-607e-49e7-8af2-b9053eda759c/volumes"
Nov 02 15:32:51.396047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:51.396011    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"]
Nov 02 15:32:51.433641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:51.433614    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"]
Nov 02 15:32:56.396459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.396414    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d" podUID="8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" containerName="payload-processing" containerID="cri-o://8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb" gracePeriod=25
Nov 02 15:32:56.434266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.434231    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" podUID="b9410559-197b-400c-9680-a61e0127df19" containerName="payload-pre-processing" containerID="cri-o://c9d7d3ddf3976cd2efb9f73d00468d6a7ed8b9904579bb1d6e3cc8e5d5306bc6" gracePeriod=25
Nov 02 15:32:56.689350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.689322    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:32:56.731626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.729803    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume\") pod \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") "
Nov 02 15:32:56.731626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.729928    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88\" (UniqueName: \"kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88\") pod \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\" (UID: \"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171\") "
Nov 02 15:32:56.731626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.730754    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume" pod "8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" (UID: "8bf019e4-176c-4e5e-bd8d-a8c7bfd55171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:56.731626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.731108    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:56.748462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.748430    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88" pod "8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" (UID: "8bf019e4-176c-4e5e-bd8d-a8c7bfd55171"). InnerVolumeSpecName "kube-api-access-fdv88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:56.832758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:56.832726    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdv88\" (UniqueName: \"kubernetes.io/projected/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171-kube-api-access-fdv88\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:32:57.421854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.421817    5041 generic.go:356] "Generic (PLEG): container finished" podID="8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" containerID="8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb" exitCode=0
Nov 02 15:32:57.422273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.421874    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"
Nov 02 15:32:57.422273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.421895    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d" event={"ID":"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171","Type":"ContainerDied","Data":"8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb"}
Nov 02 15:32:57.422273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.421945    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d" event={"ID":"8bf019e4-176c-4e5e-bd8d-a8c7bfd55171","Type":"ContainerDied","Data":"740ea4507d6dd7e38e5a6c8fed069eb5604e242120fae2d7f230c565a908de2c"}
Nov 02 15:32:57.422273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.421965    5041 scope.go:122] "RemoveContainer" containerID="8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb"
Nov 02 15:32:57.434173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.434155    5041 scope.go:122] "RemoveContainer" containerID="8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb"
Nov 02 15:32:57.434441 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:32:57.434422    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb\": container with ID starting with 8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb not found: ID does not exist" containerID="8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb"
Nov 02 15:32:57.434506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.434447    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb"} err="failed to get container status \"8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb\": rpc error: code = NotFound desc = could not find container \"8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb\": container with ID starting with 8a4fb5c93a5d3cf0678c8b872d2efe1d421bfcf1a7c50880442ac814595d93bb not found: ID does not exist"
Nov 02 15:32:57.474256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.474232    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"]
Nov 02 15:32:57.484789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.484770    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daba5-5648dcf455-zmg5d"]
Nov 02 15:32:57.890998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:32:57.890926    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" path="/var/lib/kubelet/pods/8bf019e4-176c-4e5e-bd8d-a8c7bfd55171/volumes"
Nov 02 15:33:05.750171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:05.750132    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" podUID="b9410559-197b-400c-9680-a61e0127df19" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:33:06.769508 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:33:06.769469    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:33:08.238574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:08.238535    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"]
Nov 02 15:33:08.239339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:08.239302    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy" containerID="cri-o://476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d" gracePeriod=30
Nov 02 15:33:13.545499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.545472    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:33:13.618805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.618773    5041 generic.go:356] "Generic (PLEG): container finished" podID="e46212a2-f445-4920-999b-cb38375d9927" containerID="476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d" exitCode=0
Nov 02 15:33:13.619000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.618824    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"
Nov 02 15:33:13.619000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.618878    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" event={"ID":"e46212a2-f445-4920-999b-cb38375d9927","Type":"ContainerDied","Data":"476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d"}
Nov 02 15:33:13.619000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.618917    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j" event={"ID":"e46212a2-f445-4920-999b-cb38375d9927","Type":"ContainerDied","Data":"663cb90f26d0b719f65a5e6e21ae77ce6c5e4eb3123365ab10ee782dee739900"}
Nov 02 15:33:13.619000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.618939    5041 scope.go:122] "RemoveContainer" containerID="476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d"
Nov 02 15:33:13.632226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.632201    5041 scope.go:122] "RemoveContainer" containerID="476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d"
Nov 02 15:33:13.633328 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:33:13.633291    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d\": container with ID starting with 476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d not found: ID does not exist" containerID="476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d"
Nov 02 15:33:13.633492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.633330    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d"} err="failed to get container status \"476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d\": rpc error: code = NotFound desc = could not find container \"476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d\": container with ID starting with 476348162c896217edb48a8d96224b4f2fa1d0892041360d927be96364d64e0d not found: ID does not exist"
Nov 02 15:33:13.673780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.673757    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.673939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.673812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.673939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.673840    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.673939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.673861    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.674112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.673999    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.674112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674057    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.674112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674086    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.674294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674127    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.674294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674144    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.674423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674323    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data\") pod \"e46212a2-f445-4920-999b-cb38375d9927\" (UID: \"e46212a2-f445-4920-999b-cb38375d9927\") "
Nov 02 15:33:13.674423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674347    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.674423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674266    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:13.674598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674511    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.674598 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.674581    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.675264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.675239    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.676137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.675268    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.676137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.675284    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.676137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.675299    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46212a2-f445-4920-999b-cb38375d9927-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.676137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.675311    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.676393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.676300    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.676530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.676506    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "kube-api-access-l5p4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:13.676805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.676782    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:33:13.676922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.676903    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token" pod "e46212a2-f445-4920-999b-cb38375d9927" (UID: "e46212a2-f445-4920-999b-cb38375d9927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:13.776819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.776768    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.776819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.776792    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46212a2-f445-4920-999b-cb38375d9927-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.776819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.776810    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46212a2-f445-4920-999b-cb38375d9927-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.776819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.776819    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5p4b\" (UniqueName: \"kubernetes.io/projected/e46212a2-f445-4920-999b-cb38375d9927-kube-api-access-l5p4b\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.962179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.962153    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"]
Nov 02 15:33:13.973290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:13.973262    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-jwx6j"]
Nov 02 15:33:15.750155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:15.750119    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" podUID="b9410559-197b-400c-9680-a61e0127df19" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:33:15.889770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:15.889741    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46212a2-f445-4920-999b-cb38375d9927" path="/var/lib/kubelet/pods/e46212a2-f445-4920-999b-cb38375d9927/volumes"
Nov 02 15:33:16.652464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.652430    5041 generic.go:356] "Generic (PLEG): container finished" podID="b9410559-197b-400c-9680-a61e0127df19" containerID="c9d7d3ddf3976cd2efb9f73d00468d6a7ed8b9904579bb1d6e3cc8e5d5306bc6" exitCode=0
Nov 02 15:33:16.652635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.652486    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" event={"ID":"b9410559-197b-400c-9680-a61e0127df19","Type":"ContainerDied","Data":"c9d7d3ddf3976cd2efb9f73d00468d6a7ed8b9904579bb1d6e3cc8e5d5306bc6"}
Nov 02 15:33:16.725830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.725810    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:33:16.832836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.832809    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9\" (UniqueName: \"kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9\") pod \"b9410559-197b-400c-9680-a61e0127df19\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") "
Nov 02 15:33:16.833243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.833001    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume\") pod \"b9410559-197b-400c-9680-a61e0127df19\" (UID: \"b9410559-197b-400c-9680-a61e0127df19\") "
Nov 02 15:33:16.833421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.833399    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume" pod "b9410559-197b-400c-9680-a61e0127df19" (UID: "b9410559-197b-400c-9680-a61e0127df19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:16.834873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.834850    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9" pod "b9410559-197b-400c-9680-a61e0127df19" (UID: "b9410559-197b-400c-9680-a61e0127df19"). InnerVolumeSpecName "kube-api-access-xhtc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:16.935075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.935041    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtc9\" (UniqueName: \"kubernetes.io/projected/b9410559-197b-400c-9680-a61e0127df19-kube-api-access-xhtc9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:16.935262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:16.935243    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9410559-197b-400c-9680-a61e0127df19-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:17.663732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:17.663691    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"
Nov 02 15:33:17.663920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:17.663697    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8" event={"ID":"b9410559-197b-400c-9680-a61e0127df19","Type":"ContainerDied","Data":"cbdcb7cfd7890a1ad08baf4aa2e01ac42b8938e4abaccbba1bdb539f157bdb18"}
Nov 02 15:33:17.663920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:17.663878    5041 scope.go:122] "RemoveContainer" containerID="c9d7d3ddf3976cd2efb9f73d00468d6a7ed8b9904579bb1d6e3cc8e5d5306bc6"
Nov 02 15:33:17.770746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:17.770723    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"]
Nov 02 15:33:17.779028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:17.779002    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daba5-84499f47-sq8t8"]
Nov 02 15:33:17.890110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:17.890081    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9410559-197b-400c-9680-a61e0127df19" path="/var/lib/kubelet/pods/b9410559-197b-400c-9680-a61e0127df19/volumes"
Nov 02 15:33:30.103033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.102991    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"]
Nov 02 15:33:30.103612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.103231    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd" podUID="f7831b98-9674-4352-8847-f1119617ae99" containerName="maas-api" containerID="cri-o://be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db" gracePeriod=30
Nov 02 15:33:30.484528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.484499    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:33:30.577964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.577126    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca\") pod \"f7831b98-9674-4352-8847-f1119617ae99\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") "
Nov 02 15:33:30.577964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.577271    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls\") pod \"f7831b98-9674-4352-8847-f1119617ae99\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") "
Nov 02 15:33:30.577964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.577382    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc\" (UniqueName: \"kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc\") pod \"f7831b98-9674-4352-8847-f1119617ae99\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") "
Nov 02 15:33:30.577964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.577419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs\") pod \"f7831b98-9674-4352-8847-f1119617ae99\" (UID: \"f7831b98-9674-4352-8847-f1119617ae99\") "
Nov 02 15:33:30.599391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.596531    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls" pod "f7831b98-9674-4352-8847-f1119617ae99" (UID: "f7831b98-9674-4352-8847-f1119617ae99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:33:30.599890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.599836    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca" pod "f7831b98-9674-4352-8847-f1119617ae99" (UID: "f7831b98-9674-4352-8847-f1119617ae99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:30.607388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.600493    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs" pod "f7831b98-9674-4352-8847-f1119617ae99" (UID: "f7831b98-9674-4352-8847-f1119617ae99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:33:30.626029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.620548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc" pod "f7831b98-9674-4352-8847-f1119617ae99" (UID: "f7831b98-9674-4352-8847-f1119617ae99"). InnerVolumeSpecName "kube-api-access-jvmzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:30.680678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.680583    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:30.680678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.680624    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmzc\" (UniqueName: \"kubernetes.io/projected/f7831b98-9674-4352-8847-f1119617ae99-kube-api-access-jvmzc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:30.680678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.680641    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7831b98-9674-4352-8847-f1119617ae99-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:30.680678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.680656    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7831b98-9674-4352-8847-f1119617ae99-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:33:30.781098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.781064    5041 generic.go:356] "Generic (PLEG): container finished" podID="f7831b98-9674-4352-8847-f1119617ae99" containerID="be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db" exitCode=0
Nov 02 15:33:30.781290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.781119    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"
Nov 02 15:33:30.781290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.781146    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd" event={"ID":"f7831b98-9674-4352-8847-f1119617ae99","Type":"ContainerDied","Data":"be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db"}
Nov 02 15:33:30.781290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.781209    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd" event={"ID":"f7831b98-9674-4352-8847-f1119617ae99","Type":"ContainerDied","Data":"555c51faa372df7e73d4c2c7c0bb0d213aee5ac4cb0ed3f2b674578ca752483e"}
Nov 02 15:33:30.781290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.781233    5041 scope.go:122] "RemoveContainer" containerID="be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db"
Nov 02 15:33:30.793606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.793587    5041 scope.go:122] "RemoveContainer" containerID="be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db"
Nov 02 15:33:30.793873 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:33:30.793854    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db\": container with ID starting with be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db not found: ID does not exist" containerID="be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db"
Nov 02 15:33:30.793943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.793884    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db"} err="failed to get container status \"be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db\": rpc error: code = NotFound desc = could not find container \"be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db\": container with ID starting with be0e1d63ebc35dff6881c81ca97e91c7e1c332b916d6774097a74bfad697d9db not found: ID does not exist"
Nov 02 15:33:30.867026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.866985    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"]
Nov 02 15:33:30.871902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:30.871869    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3daba5-6cb8c98757-4t2nd"]
Nov 02 15:33:31.890813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:31.890775    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7831b98-9674-4352-8847-f1119617ae99" path="/var/lib/kubelet/pods/f7831b98-9674-4352-8847-f1119617ae99/volumes"
Nov 02 15:33:36.770763 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:33:36.770726    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:33:40.826456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:33:40.826415    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" podUID="15b2f7b139a98a51bb6e53a9a5605774" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:06.770468 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:34:06.770392    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:34:17.551497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.551416    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"]
Nov 02 15:34:17.552529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552509    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy"
Nov 02 15:34:17.552600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552531    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy"
Nov 02 15:34:17.552600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552547    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy"
Nov 02 15:34:17.552600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552552    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy"
Nov 02 15:34:17.552600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552590    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cdb38061-bac4-4186-9588-0e297d26fb1e" containerName="payload-processing"
Nov 02 15:34:17.552600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552596    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb38061-bac4-4186-9588-0e297d26fb1e" containerName="payload-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552616    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" containerName="maas-api"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552622    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" containerName="maas-api"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552629    5041 cpu_manager.go:414] "containerMap: removing container" podUID="836a1569-607e-49e7-8af2-b9053eda759c" containerName="payload-pre-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552634    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836a1569-607e-49e7-8af2-b9053eda759c" containerName="payload-pre-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552673    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b9410559-197b-400c-9680-a61e0127df19" containerName="payload-pre-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552679    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9410559-197b-400c-9680-a61e0127df19" containerName="payload-pre-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552710    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f7831b98-9674-4352-8847-f1119617ae99" containerName="maas-api"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552715    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7831b98-9674-4352-8847-f1119617ae99" containerName="maas-api"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552723    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" containerName="payload-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552728    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" containerName="payload-processing"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552736    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c4af444f-93fa-4ea2-a652-fab7df58fbc2" containerName="maas-api"
Nov 02 15:34:17.552775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552741    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4af444f-93fa-4ea2-a652-fab7df58fbc2" containerName="maas-api"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552872    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9410559-197b-400c-9680-a61e0127df19" containerName="payload-pre-processing"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552883    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7831b98-9674-4352-8847-f1119617ae99" containerName="maas-api"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552890    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf019e4-176c-4e5e-bd8d-a8c7bfd55171" containerName="payload-processing"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552901    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4af444f-93fa-4ea2-a652-fab7df58fbc2" containerName="maas-api"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552912    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05f53bc-3e54-4137-a7ab-ba0b6c06858a" containerName="istio-proxy"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552922    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46212a2-f445-4920-999b-cb38375d9927" containerName="istio-proxy"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552936    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb38061-bac4-4186-9588-0e297d26fb1e" containerName="payload-processing"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552944    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3b9e9a-3d9c-4662-8026-6a93d91fbcd7" containerName="maas-api"
Nov 02 15:34:17.553550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.552953    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="836a1569-607e-49e7-8af2-b9053eda759c" containerName="payload-pre-processing"
Nov 02 15:34:17.557354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.557332    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.584434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.584402    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-78d075\"/\"default-dockercfg-svz2x\""
Nov 02 15:34:17.584604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.584402    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-78d075\"/\"auth-test-model-78d075-kserve-self-signed-certs\""
Nov 02 15:34:17.584604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.584481    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-78d075\"/\"kube-root-ca.crt\""
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.587979    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.588022    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.588046    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqsj\" (UniqueName: \"kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.588212    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.588252    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.588303    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.588516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.588466    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-78d075\"/\"openshift-service-ca.crt\""
Nov 02 15:34:17.596028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.596000    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"]
Nov 02 15:34:17.689614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.689580    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.689614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.689617    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.689838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.689638    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqsj\" (UniqueName: \"kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.689838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.689769    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.689838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.689802    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.690008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.689986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.690073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.690040    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.690073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.690058    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.690175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.690110    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.692238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.692204    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.692543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.692525    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.712235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.712211    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqsj\" (UniqueName: \"kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj\") pod \"auth-test-model-78d075-kserve-666f9469bf-nscv5\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") " pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:17.880769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:17.880609    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:18.197947 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:34:18.197888    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6e8d9ea_1522_4da9_9f84_513fc2281fee.slice/crio-ba2b2aa7fb61e61b3baebe4740d530feeaf6f79e99fa6460c2573692dca9b06b WatchSource:0}: Error finding container ba2b2aa7fb61e61b3baebe4740d530feeaf6f79e99fa6460c2573692dca9b06b: Status 404 returned error can't find the container with id ba2b2aa7fb61e61b3baebe4740d530feeaf6f79e99fa6460c2573692dca9b06b
Nov 02 15:34:18.205669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:18.205644    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"]
Nov 02 15:34:19.216846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:19.216788    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5" event={"ID":"e6e8d9ea-1522-4da9-9f84-513fc2281fee","Type":"ContainerStarted","Data":"01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923"}
Nov 02 15:34:19.216846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:19.216843    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5" event={"ID":"e6e8d9ea-1522-4da9-9f84-513fc2281fee","Type":"ContainerStarted","Data":"ba2b2aa7fb61e61b3baebe4740d530feeaf6f79e99fa6460c2573692dca9b06b"}
Nov 02 15:34:19.277010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:19.276952    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5" podStartSLOduration=2.276938222 podStartE2EDuration="2.276938222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:19.273228134 +0000 UTC m=+2311.531297584" watchObservedRunningTime="2026-11-02 15:34:19.276938222 +0000 UTC m=+2311.535007671"
Nov 02 15:34:19.890256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:19.890220    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:27.891237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:27.891206    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:27.898319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:27.898297    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:28.354310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:28.354274    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:34:36.770477 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:34:36.770432    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:34:54.815996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.815961    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"]
Nov 02 15:34:54.821239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.821217    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:54.845482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.845459    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"openshift-service-ca.crt\""
Nov 02 15:34:54.851141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.851115    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"]
Nov 02 15:34:54.856229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.856208    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"kube-root-ca.crt\""
Nov 02 15:34:54.856982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.856960    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"default-dockercfg-tbgvn\""
Nov 02 15:34:54.857232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.857213    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"auth-test-model-cce791-kserve-self-signed-certs\""
Nov 02 15:34:54.981484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.981452    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74f4\" (UniqueName: \"kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:54.981635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.981499    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:54.981635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.981521    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:54.981635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.981567    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:54.981635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.981612    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:54.981817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:54.981653    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.086502    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74f4\" (UniqueName: \"kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.086573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.086605    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.086650    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.086694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.086745    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.087928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.087426    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.095383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.091762    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.095383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.092022    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.105649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.105619    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.106776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.106752    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.114277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.114133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74f4\" (UniqueName: \"kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4\") pod \"auth-test-model-cce791-kserve-6fc997ccdf-r56cb\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") " pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.135512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.135485    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:34:55.455217 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:34:55.455187    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc44cb54a_3dfe_40e1_b210_5633bd692a7f.slice/crio-9db8fc7bc53811c4ff752ae2f52abec84cdcd6d8fbc9b9470c1e6771eeb96484 WatchSource:0}: Error finding container 9db8fc7bc53811c4ff752ae2f52abec84cdcd6d8fbc9b9470c1e6771eeb96484: Status 404 returned error can't find the container with id 9db8fc7bc53811c4ff752ae2f52abec84cdcd6d8fbc9b9470c1e6771eeb96484
Nov 02 15:34:55.456448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.456006    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"]
Nov 02 15:34:55.623854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.623811    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb" event={"ID":"c44cb54a-3dfe-40e1-b210-5633bd692a7f","Type":"ContainerStarted","Data":"1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0"}
Nov 02 15:34:55.624043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.623861    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb" event={"ID":"c44cb54a-3dfe-40e1-b210-5633bd692a7f","Type":"ContainerStarted","Data":"9db8fc7bc53811c4ff752ae2f52abec84cdcd6d8fbc9b9470c1e6771eeb96484"}
Nov 02 15:34:55.707221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:55.704546    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb" podStartSLOduration=1.704525713 podStartE2EDuration="1.704525713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:55.694029753 +0000 UTC m=+2347.952099197" watchObservedRunningTime="2026-11-02 15:34:55.704525713 +0000 UTC m=+2347.962595165"
Nov 02 15:34:56.136115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:34:56.136048    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:35:05.136133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:05.136097    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:35:05.149321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:05.149298    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:35:05.757220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:05.757170    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:35:06.769893 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:35:06.769856    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:35:36.769779 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:35:36.769692    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:35:43.593206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:43.593168    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"]
Nov 02 15:35:43.593803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:43.593442    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" containerID="cri-o://e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a" gracePeriod=30
Nov 02 15:35:44.089252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:44.089215    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"]
Nov 02 15:35:44.089447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:44.089423    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" containerID="cri-o://4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea" gracePeriod=30
Nov 02 15:35:47.313634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:47.313602    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:35:47.314042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:47.313661    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:35:47.556843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:47.556806    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:35:47.557026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:47.556856    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:35:48.899192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.899165    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:35:48.967643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.967604    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.967821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.967659    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.967821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.967747    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.967821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.967809    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.967947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.967902    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.967983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.967970    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.968050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.968024    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.968184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.968116    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.968253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.968205    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket\") pod \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\" (UID: \"9fb17794-cb62-4a6e-bec6-45bc461b2a9b\") "
Nov 02 15:35:48.968727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.968659    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:48.968996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.968738    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:48.968996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.968904    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:48.969138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.969061    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:48.969138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.969084    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:48.969138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.969098    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:48.970073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.969424    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:48.970429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.970403    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:48.970696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.970672    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:48.970824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.970800    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "kube-api-access-9jq6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:48.972158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.972134    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:48.972579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:48.972557    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo" pod "9fb17794-cb62-4a6e-bec6-45bc461b2a9b" (UID: "9fb17794-cb62-4a6e-bec6-45bc461b2a9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:35:49.072818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.071760    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.072818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.071805    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.072818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.071835    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.072818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.071851    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jq6b\" (UniqueName: \"kubernetes.io/projected/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-kube-api-access-9jq6b\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.072818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.071865    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.072818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.071880    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb17794-cb62-4a6e-bec6-45bc461b2a9b-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.116155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.116121    5041 generic.go:356] "Generic (PLEG): container finished" podID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerID="e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a" exitCode=0
Nov 02 15:35:49.116292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.116160    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"
Nov 02 15:35:49.116395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.116323    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" event={"ID":"9fb17794-cb62-4a6e-bec6-45bc461b2a9b","Type":"ContainerDied","Data":"e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a"}
Nov 02 15:35:49.116395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.116391    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx" event={"ID":"9fb17794-cb62-4a6e-bec6-45bc461b2a9b","Type":"ContainerDied","Data":"1630e2fe572e00c9dbc23264f825d3d4ebb4dc155cab19c6abc39abf60f9cdeb"}
Nov 02 15:35:49.116554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.116422    5041 scope.go:122] "RemoveContainer" containerID="e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a"
Nov 02 15:35:49.118708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.118683    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:35:49.131857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.131840    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:35:49.199321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.199291    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"]
Nov 02 15:35:49.205978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.205955    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-g44vx"]
Nov 02 15:35:49.216204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.216182    5041 scope.go:122] "RemoveContainer" containerID="e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a"
Nov 02 15:35:49.216565 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:35:49.216537    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a\": container with ID starting with e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a not found: ID does not exist" containerID="e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a"
Nov 02 15:35:49.216650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.216567    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a"} err="failed to get container status \"e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a\": rpc error: code = NotFound desc = could not find container \"e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a\": container with ID starting with e434e04271b5861a0727c8a0657cd90eb2a3e767aa6dc40d38eb19046421077a not found: ID does not exist"
Nov 02 15:35:49.369101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.369043    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:35:49.476333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476303    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476357    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476487    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476519    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476544    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476576    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476666    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:49.476724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476691    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476780    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:49.476956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476784    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476796    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:49.476956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476848    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:49.476956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476884    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo\") pod \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\" (UID: \"bdbac19b-fed9-4bcd-a66d-ef9d24babefb\") "
Nov 02 15:35:49.476956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.476922    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:49.477442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.477418    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.477442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.477444    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.477636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.477453    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.477636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.477464    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.477636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.477473    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.478620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.478600    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "kube-api-access-7rqxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:49.478914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.478892    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:49.479136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.479116    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:35:49.479199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.479122    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token" pod "bdbac19b-fed9-4bcd-a66d-ef9d24babefb" (UID: "bdbac19b-fed9-4bcd-a66d-ef9d24babefb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:49.578865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.578831    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqxj\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-kube-api-access-7rqxj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.578865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.578859    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.578865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.578870    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.579091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.578883    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbac19b-fed9-4bcd-a66d-ef9d24babefb-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:35:49.891919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:49.891830    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" path="/var/lib/kubelet/pods/9fb17794-cb62-4a6e-bec6-45bc461b2a9b/volumes"
Nov 02 15:35:50.127824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.127792    5041 generic.go:356] "Generic (PLEG): container finished" podID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerID="4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea" exitCode=0
Nov 02 15:35:50.128262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.127838    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"
Nov 02 15:35:50.128262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.127850    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" event={"ID":"bdbac19b-fed9-4bcd-a66d-ef9d24babefb","Type":"ContainerDied","Data":"4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea"}
Nov 02 15:35:50.128262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.127889    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8" event={"ID":"bdbac19b-fed9-4bcd-a66d-ef9d24babefb","Type":"ContainerDied","Data":"43cf3d1d3b02587b03d8afe630c12837b2a1492dc70755a39f859efd3fd29d58"}
Nov 02 15:35:50.128262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.127911    5041 scope.go:122] "RemoveContainer" containerID="4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea"
Nov 02 15:35:50.139936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.139914    5041 scope.go:122] "RemoveContainer" containerID="4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea"
Nov 02 15:35:50.140211 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:35:50.140181    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea\": container with ID starting with 4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea not found: ID does not exist" containerID="4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea"
Nov 02 15:35:50.140266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.140219    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea"} err="failed to get container status \"4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea\": rpc error: code = NotFound desc = could not find container \"4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea\": container with ID starting with 4a2825f676b0229418ec16d034c99ea733e50fcb78983e19bda446ec78ebdaea not found: ID does not exist"
Nov 02 15:35:50.199069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.197953    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"]
Nov 02 15:35:50.204637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:50.204612    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-8v9n8"]
Nov 02 15:35:51.891060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:51.890983    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" path="/var/lib/kubelet/pods/bdbac19b-fed9-4bcd-a66d-ef9d24babefb/volumes"
Nov 02 15:35:59.188189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:35:59.188150    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"]
Nov 02 15:36:04.189900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:04.189861    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerName="payload-pre-processing" containerID="cri-o://8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0" gracePeriod=25
Nov 02 15:36:06.770265 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:36:06.770223    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:36:07.388558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:07.388504    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:17.364017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:17.363976    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:22.279616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.279580    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"]
Nov 02 15:36:22.280216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.279831    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6" podUID="e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" containerName="maas-api" containerID="cri-o://27e4cb46196784daccee9cf34af3ba7ac98df585a5da9648588916a60a4ff5a9" gracePeriod=30
Nov 02 15:36:22.551489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.548706    5041 generic.go:356] "Generic (PLEG): container finished" podID="e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" containerID="27e4cb46196784daccee9cf34af3ba7ac98df585a5da9648588916a60a4ff5a9" exitCode=0
Nov 02 15:36:22.551489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.548938    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6" event={"ID":"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e","Type":"ContainerDied","Data":"27e4cb46196784daccee9cf34af3ba7ac98df585a5da9648588916a60a4ff5a9"}
Nov 02 15:36:22.606934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.606905    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:36:22.699896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.699852    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs\") pod \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") "
Nov 02 15:36:22.700210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.699984    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\") pod \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") "
Nov 02 15:36:22.700210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.700041    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca\") pod \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") "
Nov 02 15:36:22.700210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.700151    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw\" (UniqueName: \"kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw\") pod \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\" (UID: \"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e\") "
Nov 02 15:36:22.712625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.712548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca" pod "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" (UID: "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:22.734958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.734919    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls" pod "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" (UID: "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:22.749092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.749046    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs" pod "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" (UID: "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:22.754611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.754565    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw" pod "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" (UID: "e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e"). InnerVolumeSpecName "kube-api-access-pfdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:22.802546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.802475    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:22.802546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.802509    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:22.802546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.802526    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdgw\" (UniqueName: \"kubernetes.io/projected/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-kube-api-access-pfdgw\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:22.802546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:22.802542    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:23.559186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:23.559155    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"
Nov 02 15:36:23.559627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:23.559206    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6" event={"ID":"e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e","Type":"ContainerDied","Data":"6ffa20cbad9dab01d16bb14275047901c76e26419f85c132e7f64e025ca736f4"}
Nov 02 15:36:23.559627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:23.559270    5041 scope.go:122] "RemoveContainer" containerID="27e4cb46196784daccee9cf34af3ba7ac98df585a5da9648588916a60a4ff5a9"
Nov 02 15:36:23.616495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:23.616470    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"]
Nov 02 15:36:23.623915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:23.623895    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28f621-7857694587-d8zj6"]
Nov 02 15:36:23.889483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:23.889404    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" path="/var/lib/kubelet/pods/e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e/volumes"
Nov 02 15:36:24.481790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.481765    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:36:24.525523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.525489    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp\" (UniqueName: \"kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp\") pod \"f4bf93b2-7a33-4e17-9b79-53328a28761d\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") "
Nov 02 15:36:24.525653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.525583    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume\") pod \"f4bf93b2-7a33-4e17-9b79-53328a28761d\" (UID: \"f4bf93b2-7a33-4e17-9b79-53328a28761d\") "
Nov 02 15:36:24.526584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.526556    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume" pod "f4bf93b2-7a33-4e17-9b79-53328a28761d" (UID: "f4bf93b2-7a33-4e17-9b79-53328a28761d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:24.531549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.531521    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp" pod "f4bf93b2-7a33-4e17-9b79-53328a28761d" (UID: "f4bf93b2-7a33-4e17-9b79-53328a28761d"). InnerVolumeSpecName "kube-api-access-7bgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:24.575254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.575221    5041 generic.go:356] "Generic (PLEG): container finished" podID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerID="8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0" exitCode=0
Nov 02 15:36:24.575720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.575282    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"
Nov 02 15:36:24.575720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.575421    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" event={"ID":"f4bf93b2-7a33-4e17-9b79-53328a28761d","Type":"ContainerDied","Data":"8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0"}
Nov 02 15:36:24.575720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.575475    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms" event={"ID":"f4bf93b2-7a33-4e17-9b79-53328a28761d","Type":"ContainerDied","Data":"1d8510baedf6ca0262bd4a241d5b8bfa8f0b8051479b5fe1f1ec21499467b1ab"}
Nov 02 15:36:24.575720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.575507    5041 scope.go:122] "RemoveContainer" containerID="8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0"
Nov 02 15:36:24.592478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.592455    5041 scope.go:122] "RemoveContainer" containerID="8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0"
Nov 02 15:36:24.592804 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:36:24.592785    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0\": container with ID starting with 8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0 not found: ID does not exist" containerID="8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0"
Nov 02 15:36:24.592850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.592813    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0"} err="failed to get container status \"8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0\": rpc error: code = NotFound desc = could not find container \"8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0\": container with ID starting with 8d5d3546a00f35bed2fd80a11b34a608a7c00cd5e5bf6c1fb1deab88d5af34e0 not found: ID does not exist"
Nov 02 15:36:24.628000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.627973    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bgcp\" (UniqueName: \"kubernetes.io/projected/f4bf93b2-7a33-4e17-9b79-53328a28761d-kube-api-access-7bgcp\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:24.628000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.628000    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bf93b2-7a33-4e17-9b79-53328a28761d-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:24.629151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.629128    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"]
Nov 02 15:36:24.633917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:24.633895    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-28f621-7746b5dfdc-8qwms"]
Nov 02 15:36:25.890763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:25.890730    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" path="/var/lib/kubelet/pods/f4bf93b2-7a33-4e17-9b79-53328a28761d/volumes"
Nov 02 15:36:36.770975 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:36:36.770936    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:36:42.877547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:42.877504    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"]
Nov 02 15:36:42.920525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:42.920475    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"]
Nov 02 15:36:47.878269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:47.878211    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerName="payload-processing" containerID="cri-o://2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e" gracePeriod=25
Nov 02 15:36:47.921178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:47.921129    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" containerName="payload-pre-processing" containerID="cri-o://4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906" gracePeriod=25
Nov 02 15:36:49.898864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:49.898819    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:49.899509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:49.899286    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:52.546387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:52.546331    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"]
Nov 02 15:36:52.585100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:52.585059    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"]
Nov 02 15:36:55.897179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:55.897142    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"]
Nov 02 15:36:55.897831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:55.897406    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c" podUID="734bf2eb-db6f-4808-b21e-6d1bfb141987" containerName="maas-api" containerID="cri-o://141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1" gracePeriod=30
Nov 02 15:36:56.209184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.207417    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"]
Nov 02 15:36:56.264393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.262269    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:36:56.351130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.351090    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls\") pod \"734bf2eb-db6f-4808-b21e-6d1bfb141987\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") "
Nov 02 15:36:56.351392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.351187    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk\" (UniqueName: \"kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk\") pod \"734bf2eb-db6f-4808-b21e-6d1bfb141987\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") "
Nov 02 15:36:56.351392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.351348    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca\") pod \"734bf2eb-db6f-4808-b21e-6d1bfb141987\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") "
Nov 02 15:36:56.351621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.351581    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs\") pod \"734bf2eb-db6f-4808-b21e-6d1bfb141987\" (UID: \"734bf2eb-db6f-4808-b21e-6d1bfb141987\") "
Nov 02 15:36:56.352843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.352810    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca" pod "734bf2eb-db6f-4808-b21e-6d1bfb141987" (UID: "734bf2eb-db6f-4808-b21e-6d1bfb141987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:56.355019 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.354698    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk" pod "734bf2eb-db6f-4808-b21e-6d1bfb141987" (UID: "734bf2eb-db6f-4808-b21e-6d1bfb141987"). InnerVolumeSpecName "kube-api-access-gtbwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:56.356049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.356025    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls" pod "734bf2eb-db6f-4808-b21e-6d1bfb141987" (UID: "734bf2eb-db6f-4808-b21e-6d1bfb141987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:56.356526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.356497    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs" pod "734bf2eb-db6f-4808-b21e-6d1bfb141987" (UID: "734bf2eb-db6f-4808-b21e-6d1bfb141987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:56.483451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.463745    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:56.483451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.463791    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtbwk\" (UniqueName: \"kubernetes.io/projected/734bf2eb-db6f-4808-b21e-6d1bfb141987-kube-api-access-gtbwk\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:56.483451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.463806    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734bf2eb-db6f-4808-b21e-6d1bfb141987-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:56.483451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.463821    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734bf2eb-db6f-4808-b21e-6d1bfb141987-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:36:56.889498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.889397    5041 generic.go:356] "Generic (PLEG): container finished" podID="734bf2eb-db6f-4808-b21e-6d1bfb141987" containerID="141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1" exitCode=0
Nov 02 15:36:56.889671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.889556    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c" event={"ID":"734bf2eb-db6f-4808-b21e-6d1bfb141987","Type":"ContainerDied","Data":"141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1"}
Nov 02 15:36:56.889671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.889597    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c" event={"ID":"734bf2eb-db6f-4808-b21e-6d1bfb141987","Type":"ContainerDied","Data":"fc50abb84d3c1fc9129935ea6585027b3fa61a4b74937de3c57dd783948167f3"}
Nov 02 15:36:56.889671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.889626    5041 scope.go:122] "RemoveContainer" containerID="141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1"
Nov 02 15:36:56.889841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.889802    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"
Nov 02 15:36:56.906594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.906324    5041 scope.go:122] "RemoveContainer" containerID="141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1"
Nov 02 15:36:56.906861 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:36:56.906727    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1\": container with ID starting with 141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1 not found: ID does not exist" containerID="141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1"
Nov 02 15:36:56.906861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.906756    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1"} err="failed to get container status \"141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1\": rpc error: code = NotFound desc = could not find container \"141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1\": container with ID starting with 141b82e954332485a76fc0db84f32828418ceb58a103c333b2756584aa6027a1 not found: ID does not exist"
Nov 02 15:36:56.950893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.950860    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"]
Nov 02 15:36:56.956493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:56.956469    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f8b7c-dcdb6c87b-x7r6c"]
Nov 02 15:36:57.546858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:57.546812    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerName="payload-processing" containerID="cri-o://a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae" gracePeriod=25
Nov 02 15:36:57.839895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:57.839812    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" containerName="payload-pre-processing" containerID="cri-o://ab6cd1a2c63f9da88223746aa5dc123918f108eb9b0d59362a919d4a15f32f16" gracePeriod=25
Nov 02 15:36:57.890959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:57.890917    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734bf2eb-db6f-4808-b21e-6d1bfb141987" path="/var/lib/kubelet/pods/734bf2eb-db6f-4808-b21e-6d1bfb141987/volumes"
Nov 02 15:36:59.881087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:59.880987    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:59.881087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:36:59.881015    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:01.208092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:01.208052    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerName="payload-pre-processing" containerID="cri-o://853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f" gracePeriod=25
Nov 02 15:37:05.955123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:05.955081    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:06.770559 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:06.770521    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:37:07.268590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:07.268547    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:07.282607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:07.282569    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:08.218607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.218574    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:37:08.253328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.253301    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:37:08.351977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.351947    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545\" (UniqueName: \"kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545\") pod \"75824f88-123f-4256-8369-ccb1efcbaa50\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") "
Nov 02 15:37:08.352482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.352153    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume\") pod \"687dc132-d1be-4d39-aa20-873e5a4191f0\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") "
Nov 02 15:37:08.352482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.352259    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq\" (UniqueName: \"kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq\") pod \"687dc132-d1be-4d39-aa20-873e5a4191f0\" (UID: \"687dc132-d1be-4d39-aa20-873e5a4191f0\") "
Nov 02 15:37:08.352482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.352330    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume\") pod \"75824f88-123f-4256-8369-ccb1efcbaa50\" (UID: \"75824f88-123f-4256-8369-ccb1efcbaa50\") "
Nov 02 15:37:08.352664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.352593    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume" pod "687dc132-d1be-4d39-aa20-873e5a4191f0" (UID: "687dc132-d1be-4d39-aa20-873e5a4191f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:08.352737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.352706    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume" pod "75824f88-123f-4256-8369-ccb1efcbaa50" (UID: "75824f88-123f-4256-8369-ccb1efcbaa50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:08.353090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.353069    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687dc132-d1be-4d39-aa20-873e5a4191f0-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.353139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.353097    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75824f88-123f-4256-8369-ccb1efcbaa50-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.354074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.354051    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545" pod "75824f88-123f-4256-8369-ccb1efcbaa50" (UID: "75824f88-123f-4256-8369-ccb1efcbaa50"). InnerVolumeSpecName "kube-api-access-64545". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:08.354398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.354379    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq" pod "687dc132-d1be-4d39-aa20-873e5a4191f0" (UID: "687dc132-d1be-4d39-aa20-873e5a4191f0"). InnerVolumeSpecName "kube-api-access-jjqbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:08.454563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.454537    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjqbq\" (UniqueName: \"kubernetes.io/projected/687dc132-d1be-4d39-aa20-873e5a4191f0-kube-api-access-jjqbq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.454700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.454566    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64545\" (UniqueName: \"kubernetes.io/projected/75824f88-123f-4256-8369-ccb1efcbaa50-kube-api-access-64545\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.991238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.991200    5041 generic.go:356] "Generic (PLEG): container finished" podID="75824f88-123f-4256-8369-ccb1efcbaa50" containerID="4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906" exitCode=0
Nov 02 15:37:08.993435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.991502    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"
Nov 02 15:37:08.993435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.991756    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" event={"ID":"75824f88-123f-4256-8369-ccb1efcbaa50","Type":"ContainerDied","Data":"4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906"}
Nov 02 15:37:08.993435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.991809    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b" event={"ID":"75824f88-123f-4256-8369-ccb1efcbaa50","Type":"ContainerDied","Data":"9dda5a5ca41c519e19f0a89a0db9410e0818b2979ac616e6ad56fa9ab71d3fbe"}
Nov 02 15:37:08.993435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.991837    5041 scope.go:122] "RemoveContainer" containerID="4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906"
Nov 02 15:37:08.999649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.999620    5041 generic.go:356] "Generic (PLEG): container finished" podID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerID="2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e" exitCode=0
Nov 02 15:37:08.999815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.999795    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" event={"ID":"687dc132-d1be-4d39-aa20-873e5a4191f0","Type":"ContainerDied","Data":"2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e"}
Nov 02 15:37:09.000003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:08.999983    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c" event={"ID":"687dc132-d1be-4d39-aa20-873e5a4191f0","Type":"ContainerDied","Data":"0ccd8e02d754eb764dca5185a997732e22cbbde76d7a706a240f40d382511241"}
Nov 02 15:37:09.000187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.000170    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"
Nov 02 15:37:09.016033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.016015    5041 scope.go:122] "RemoveContainer" containerID="4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906"
Nov 02 15:37:09.016296 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:09.016274    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906\": container with ID starting with 4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906 not found: ID does not exist" containerID="4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906"
Nov 02 15:37:09.016408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.016300    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906"} err="failed to get container status \"4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906\": rpc error: code = NotFound desc = could not find container \"4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906\": container with ID starting with 4a7f281c681f67125f05e2c05e3a009a85f5e83f37a2dc1b19f57030571bf906 not found: ID does not exist"
Nov 02 15:37:09.016408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.016317    5041 scope.go:122] "RemoveContainer" containerID="2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e"
Nov 02 15:37:09.028695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.028544    5041 scope.go:122] "RemoveContainer" containerID="2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e"
Nov 02 15:37:09.028829 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:09.028805    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e\": container with ID starting with 2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e not found: ID does not exist" containerID="2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e"
Nov 02 15:37:09.028879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.028839    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e"} err="failed to get container status \"2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e\": rpc error: code = NotFound desc = could not find container \"2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e\": container with ID starting with 2f3edb35c957123b9ae83321782efd0d0127eca0b68acccb6b3722e52588a59e not found: ID does not exist"
Nov 02 15:37:09.068761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.068733    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"]
Nov 02 15:37:09.075435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.075412    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cfc63a-74bc49c65f-qwk6c"]
Nov 02 15:37:09.141918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.141883    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"]
Nov 02 15:37:09.142156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.142137    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cfc63a-645668dfb7-9pr4b"]
Nov 02 15:37:09.889146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.889098    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" path="/var/lib/kubelet/pods/687dc132-d1be-4d39-aa20-873e5a4191f0/volumes"
Nov 02 15:37:09.889730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:09.889708    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" path="/var/lib/kubelet/pods/75824f88-123f-4256-8369-ccb1efcbaa50/volumes"
Nov 02 15:37:10.263449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:10.263415    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"]
Nov 02 15:37:10.263908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:10.263869    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy" containerID="cri-o://64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d" gracePeriod=30
Nov 02 15:37:10.274202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:10.274175    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"]
Nov 02 15:37:10.274481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:10.274356    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" containerID="cri-o://5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b" gracePeriod=30
Nov 02 15:37:11.042832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.042799    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"]
Nov 02 15:37:11.043633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.043587    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj" podUID="55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" containerName="maas-api" containerID="cri-o://be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b" gracePeriod=30
Nov 02 15:37:11.368082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.368054    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:37:11.510887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.510854    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca\") pod \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") "
Nov 02 15:37:11.511090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.510977    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24\" (UniqueName: \"kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24\") pod \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") "
Nov 02 15:37:11.511090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.511034    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls\") pod \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") "
Nov 02 15:37:11.511090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.511056    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs\") pod \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\" (UID: \"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5\") "
Nov 02 15:37:11.511537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.511502    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca" pod "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" (UID: "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:11.513685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.513651    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24" pod "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" (UID: "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5"). InnerVolumeSpecName "kube-api-access-fjp24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:11.513865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.513842    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs" pod "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" (UID: "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:11.515500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.515475    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls" pod "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" (UID: "55ad83f6-7c13-4908-b2cb-9ffc34f39ce5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:11.613430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.613345    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:11.613430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.613388    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjp24\" (UniqueName: \"kubernetes.io/projected/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-kube-api-access-fjp24\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:11.613430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.613400    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:11.613430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:11.613410    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:12.027772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.027732    5041 generic.go:356] "Generic (PLEG): container finished" podID="55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" containerID="be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b" exitCode=0
Nov 02 15:37:12.027961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.027789    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"
Nov 02 15:37:12.027961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.027812    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj" event={"ID":"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5","Type":"ContainerDied","Data":"be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b"}
Nov 02 15:37:12.027961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.027867    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj" event={"ID":"55ad83f6-7c13-4908-b2cb-9ffc34f39ce5","Type":"ContainerDied","Data":"a0f3eb4f184d855897bad0fb4679590d2f7b1688f2bf5772d9dcf4d55fe33078"}
Nov 02 15:37:12.027961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.027889    5041 scope.go:122] "RemoveContainer" containerID="be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b"
Nov 02 15:37:12.042676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.042652    5041 scope.go:122] "RemoveContainer" containerID="be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b"
Nov 02 15:37:12.043192 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:12.043164    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b\": container with ID starting with be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b not found: ID does not exist" containerID="be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b"
Nov 02 15:37:12.043570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.043197    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b"} err="failed to get container status \"be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b\": rpc error: code = NotFound desc = could not find container \"be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b\": container with ID starting with be1d3d70fa9650a860ef746710dcb6189b4a9ebbd62dee4c1099fe37b003043b not found: ID does not exist"
Nov 02 15:37:12.082393 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.082360    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"]
Nov 02 15:37:12.086660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:12.086639    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80ec03-7464c8b897-lczjj"]
Nov 02 15:37:13.891039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:13.891004    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" path="/var/lib/kubelet/pods/55ad83f6-7c13-4908-b2cb-9ffc34f39ce5/volumes"
Nov 02 15:37:14.957848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:14.957813    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jlsxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:37:14.958226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:14.957870    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:37:15.605765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.605724    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:37:15.644262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.644236    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:37:15.809952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.809858    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.809952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.809927    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.810171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.809976    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.810171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.810018    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.810171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.810122    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.810171 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.810151    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.810431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.810226    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.810431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.810268    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.811420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811347    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811461    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811529    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811572    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811620    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811683    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811778    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket\") pod \"6c8c355a-4867-4eee-98a4-0ccca21f6301\" (UID: \"6c8c355a-4867-4eee-98a4-0ccca21f6301\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811845    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811923    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.812343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811980    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket\") pod \"417d8860-2269-40da-acb1-bccef5de32e9\" (UID: \"417d8860-2269-40da-acb1-bccef5de32e9\") "
Nov 02 15:37:15.814176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.810610    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.811244    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.812261    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.812498    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.813711    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.813940    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.814037    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.814142    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.814689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.814398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:15.814750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.814722    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.816568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.816529    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:15.817274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.817227    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "kube-api-access-x7qhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:15.819629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.819593    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:15.819741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.819672    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.819815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.819784    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:15.820043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.820006    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "kube-api-access-xbq7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:15.820281 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.820258    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token" pod "6c8c355a-4867-4eee-98a4-0ccca21f6301" (UID: "6c8c355a-4867-4eee-98a4-0ccca21f6301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:15.821217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.821194    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token" pod "417d8860-2269-40da-acb1-bccef5de32e9" (UID: "417d8860-2269-40da-acb1-bccef5de32e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:15.920265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.920230    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921471    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7qhq\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-kube-api-access-x7qhq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921502    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921518    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921532    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921545    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8c355a-4867-4eee-98a4-0ccca21f6301-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921558    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921571    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921585    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921598    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921611    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbq7h\" (UniqueName: \"kubernetes.io/projected/417d8860-2269-40da-acb1-bccef5de32e9-kube-api-access-xbq7h\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921626    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417d8860-2269-40da-acb1-bccef5de32e9-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921639    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921652    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8c355a-4867-4eee-98a4-0ccca21f6301-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.921660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921665    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417d8860-2269-40da-acb1-bccef5de32e9-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.922152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921677    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.922152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921690    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417d8860-2269-40da-acb1-bccef5de32e9-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.922152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.921703    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8c355a-4867-4eee-98a4-0ccca21f6301-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:15.955482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:15.955447    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:16.062891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.062825    5041 generic.go:356] "Generic (PLEG): container finished" podID="417d8860-2269-40da-acb1-bccef5de32e9" containerID="5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b" exitCode=0
Nov 02 15:37:16.063294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.062888    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"
Nov 02 15:37:16.063294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.062913    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" event={"ID":"417d8860-2269-40da-acb1-bccef5de32e9","Type":"ContainerDied","Data":"5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b"}
Nov 02 15:37:16.063294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.062970    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv" event={"ID":"417d8860-2269-40da-acb1-bccef5de32e9","Type":"ContainerDied","Data":"dfed9fd56eb3fcd1348cafdba5ee4d2412d3c59f4578728af76428e4b5bd1ebb"}
Nov 02 15:37:16.063294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.063002    5041 scope.go:122] "RemoveContainer" containerID="5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b"
Nov 02 15:37:16.064886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.064858    5041 generic.go:356] "Generic (PLEG): container finished" podID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerID="64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d" exitCode=0
Nov 02 15:37:16.065029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.065010    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"
Nov 02 15:37:16.065104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.065059    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" event={"ID":"6c8c355a-4867-4eee-98a4-0ccca21f6301","Type":"ContainerDied","Data":"64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d"}
Nov 02 15:37:16.065167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.065105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb" event={"ID":"6c8c355a-4867-4eee-98a4-0ccca21f6301","Type":"ContainerDied","Data":"c2fd6ca4502596df8dfc513c4660e40cf30aac2bfe004c0c02aa8b2a5dea0bf4"}
Nov 02 15:37:16.076144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.076113    5041 scope.go:122] "RemoveContainer" containerID="5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b"
Nov 02 15:37:16.076403 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:16.076382    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b\": container with ID starting with 5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b not found: ID does not exist" containerID="5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b"
Nov 02 15:37:16.076486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.076409    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b"} err="failed to get container status \"5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b\": rpc error: code = NotFound desc = could not find container \"5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b\": container with ID starting with 5e5060a07cbc3316cae2da530a987f1f745fcc3568e89d7cbf1ab12a4af8be8b not found: ID does not exist"
Nov 02 15:37:16.076486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.076426    5041 scope.go:122] "RemoveContainer" containerID="64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d"
Nov 02 15:37:16.086450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.086428    5041 scope.go:122] "RemoveContainer" containerID="64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d"
Nov 02 15:37:16.086729 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:16.086710    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d\": container with ID starting with 64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d not found: ID does not exist" containerID="64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d"
Nov 02 15:37:16.086783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.086736    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d"} err="failed to get container status \"64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d\": rpc error: code = NotFound desc = could not find container \"64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d\": container with ID starting with 64b6a02b73cefa0656c8fbad1bc9f019e01b6474225e8ff08a018f6bacb84e1d not found: ID does not exist"
Nov 02 15:37:16.118645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.118619    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"]
Nov 02 15:37:16.123132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.123108    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8q6sb"]
Nov 02 15:37:16.155297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.155276    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"]
Nov 02 15:37:16.161148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:16.161127    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jlsxv"]
Nov 02 15:37:17.268452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.268410    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:17.282048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.282007    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:17.829728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.829646    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:37:17.858946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.857976    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722\" (UniqueName: \"kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722\") pod \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") "
Nov 02 15:37:17.858946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.858090    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume\") pod \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\" (UID: \"4a621c27-1503-48ee-a17b-f25f9d4f0b3a\") "
Nov 02 15:37:17.859693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.859328    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume" pod "4a621c27-1503-48ee-a17b-f25f9d4f0b3a" (UID: "4a621c27-1503-48ee-a17b-f25f9d4f0b3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:17.862553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.862517    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722" pod "4a621c27-1503-48ee-a17b-f25f9d4f0b3a" (UID: "4a621c27-1503-48ee-a17b-f25f9d4f0b3a"). InnerVolumeSpecName "kube-api-access-8g722". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:17.894442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.894414    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417d8860-2269-40da-acb1-bccef5de32e9" path="/var/lib/kubelet/pods/417d8860-2269-40da-acb1-bccef5de32e9/volumes"
Nov 02 15:37:17.895098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.895073    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" path="/var/lib/kubelet/pods/6c8c355a-4867-4eee-98a4-0ccca21f6301/volumes"
Nov 02 15:37:17.960491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.960399    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g722\" (UniqueName: \"kubernetes.io/projected/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-kube-api-access-8g722\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:17.960491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:17.960433    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a621c27-1503-48ee-a17b-f25f9d4f0b3a-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:18.087305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.086736    5041 generic.go:356] "Generic (PLEG): container finished" podID="31e4fb40-3326-425b-be8c-40c03199cf10" containerID="ab6cd1a2c63f9da88223746aa5dc123918f108eb9b0d59362a919d4a15f32f16" exitCode=0
Nov 02 15:37:18.087305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.086926    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" event={"ID":"31e4fb40-3326-425b-be8c-40c03199cf10","Type":"ContainerDied","Data":"ab6cd1a2c63f9da88223746aa5dc123918f108eb9b0d59362a919d4a15f32f16"}
Nov 02 15:37:18.088610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.088579    5041 generic.go:356] "Generic (PLEG): container finished" podID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerID="a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae" exitCode=0
Nov 02 15:37:18.088716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.088643    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" event={"ID":"4a621c27-1503-48ee-a17b-f25f9d4f0b3a","Type":"ContainerDied","Data":"a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae"}
Nov 02 15:37:18.088716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.088687    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99" event={"ID":"4a621c27-1503-48ee-a17b-f25f9d4f0b3a","Type":"ContainerDied","Data":"7e764e6056e6bbcca1f0c5ce9a415bfd55f2f50966aeb728e530eb1a275a59fa"}
Nov 02 15:37:18.088844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.088721    5041 scope.go:122] "RemoveContainer" containerID="a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae"
Nov 02 15:37:18.088901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.088879    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"
Nov 02 15:37:18.102235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.102209    5041 scope.go:122] "RemoveContainer" containerID="a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae"
Nov 02 15:37:18.102576 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:18.102552    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae\": container with ID starting with a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae not found: ID does not exist" containerID="a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae"
Nov 02 15:37:18.102680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.102583    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae"} err="failed to get container status \"a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae\": rpc error: code = NotFound desc = could not find container \"a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae\": container with ID starting with a73232a7df4640f6ba31c2ef9d490516faf86c1f3ac202ba8b149a52b4647bae not found: ID does not exist"
Nov 02 15:37:18.146654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.146628    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:37:18.148196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.147789    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"]
Nov 02 15:37:18.154663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.154635    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f8b7c-6bf64979bf-2kr99"]
Nov 02 15:37:18.264802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.264770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl\" (UniqueName: \"kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl\") pod \"31e4fb40-3326-425b-be8c-40c03199cf10\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") "
Nov 02 15:37:18.264978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.264844    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume\") pod \"31e4fb40-3326-425b-be8c-40c03199cf10\" (UID: \"31e4fb40-3326-425b-be8c-40c03199cf10\") "
Nov 02 15:37:18.265233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.265207    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume" pod "31e4fb40-3326-425b-be8c-40c03199cf10" (UID: "31e4fb40-3326-425b-be8c-40c03199cf10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:18.267042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.267016    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl" pod "31e4fb40-3326-425b-be8c-40c03199cf10" (UID: "31e4fb40-3326-425b-be8c-40c03199cf10"). InnerVolumeSpecName "kube-api-access-mflgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:18.366260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.366179    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mflgl\" (UniqueName: \"kubernetes.io/projected/31e4fb40-3326-425b-be8c-40c03199cf10-kube-api-access-mflgl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:18.366260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:18.366211    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e4fb40-3326-425b-be8c-40c03199cf10-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:19.098816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.098783    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"
Nov 02 15:37:19.099014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.098818    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67" event={"ID":"31e4fb40-3326-425b-be8c-40c03199cf10","Type":"ContainerDied","Data":"f4d4e88a4b113ad21bcd78de07687073528461e2b703d730c8ceac8ba00a26ff"}
Nov 02 15:37:19.099014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.098873    5041 scope.go:122] "RemoveContainer" containerID="ab6cd1a2c63f9da88223746aa5dc123918f108eb9b0d59362a919d4a15f32f16"
Nov 02 15:37:19.153207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.153179    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"]
Nov 02 15:37:19.182095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.182058    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f8b7c-66bcc9c447-56g67"]
Nov 02 15:37:19.895923 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.895887    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" path="/var/lib/kubelet/pods/31e4fb40-3326-425b-be8c-40c03199cf10/volumes"
Nov 02 15:37:19.897112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:19.897081    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" path="/var/lib/kubelet/pods/4a621c27-1503-48ee-a17b-f25f9d4f0b3a/volumes"
Nov 02 15:37:21.500152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.500123    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:37:21.621349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.621309    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume\") pod \"738bb0ae-3436-4de8-9cbf-30a55e30a104\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") "
Nov 02 15:37:21.621723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.621695    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc\" (UniqueName: \"kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc\") pod \"738bb0ae-3436-4de8-9cbf-30a55e30a104\" (UID: \"738bb0ae-3436-4de8-9cbf-30a55e30a104\") "
Nov 02 15:37:21.621901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.621786    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume" pod "738bb0ae-3436-4de8-9cbf-30a55e30a104" (UID: "738bb0ae-3436-4de8-9cbf-30a55e30a104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:21.622292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.622266    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738bb0ae-3436-4de8-9cbf-30a55e30a104-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:21.624075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.624044    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc" pod "738bb0ae-3436-4de8-9cbf-30a55e30a104" (UID: "738bb0ae-3436-4de8-9cbf-30a55e30a104"). InnerVolumeSpecName "kube-api-access-wl5kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:21.723423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:21.723353    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5kc\" (UniqueName: \"kubernetes.io/projected/738bb0ae-3436-4de8-9cbf-30a55e30a104-kube-api-access-wl5kc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:22.128269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.128193    5041 generic.go:356] "Generic (PLEG): container finished" podID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerID="853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f" exitCode=0
Nov 02 15:37:22.128269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.128235    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"
Nov 02 15:37:22.128486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.128283    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" event={"ID":"738bb0ae-3436-4de8-9cbf-30a55e30a104","Type":"ContainerDied","Data":"853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f"}
Nov 02 15:37:22.128486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.128338    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb" event={"ID":"738bb0ae-3436-4de8-9cbf-30a55e30a104","Type":"ContainerDied","Data":"b6b001008fd6d01b21888caa4fdeab24f18503c6ca956115578b8962c9b6d262"}
Nov 02 15:37:22.128486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.128396    5041 scope.go:122] "RemoveContainer" containerID="853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f"
Nov 02 15:37:22.139391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.139347    5041 scope.go:122] "RemoveContainer" containerID="853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f"
Nov 02 15:37:22.139658 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:22.139636    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f\": container with ID starting with 853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f not found: ID does not exist" containerID="853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f"
Nov 02 15:37:22.139765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.139662    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f"} err="failed to get container status \"853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f\": rpc error: code = NotFound desc = could not find container \"853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f\": container with ID starting with 853451d77a30d30eb31ba82c02e1a20fa5e26a6e949feb573b9e1a29266e043f not found: ID does not exist"
Nov 02 15:37:22.183147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.183127    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"]
Nov 02 15:37:22.194439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:22.194415    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80ec03-745b6877ff-77lfb"]
Nov 02 15:37:23.899549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:23.899516    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" path="/var/lib/kubelet/pods/738bb0ae-3436-4de8-9cbf-30a55e30a104/volumes"
Nov 02 15:37:27.025577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:27.025542    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"]
Nov 02 15:37:32.169445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:32.169389    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" containerName="payload-pre-processing" containerID="cri-o://6e75610bcef7c1b867aa8e3d76b053448acebd593bece6fa98f316e350c0c868" gracePeriod=25
Nov 02 15:37:34.823690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:34.823655    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"]
Nov 02 15:37:34.824112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:34.823904    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9" podUID="f206e5b3-c341-47ea-ade7-31819dc76c46" containerName="maas-api" containerID="cri-o://b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9" gracePeriod=30
Nov 02 15:37:35.140269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.140232    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:37:35.202477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.198424    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca\") pod \"f206e5b3-c341-47ea-ade7-31819dc76c46\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") "
Nov 02 15:37:35.202477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.198887    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls\") pod \"f206e5b3-c341-47ea-ade7-31819dc76c46\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") "
Nov 02 15:37:35.202477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.199101    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t\" (UniqueName: \"kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t\") pod \"f206e5b3-c341-47ea-ade7-31819dc76c46\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") "
Nov 02 15:37:35.202477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.199164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs\") pod \"f206e5b3-c341-47ea-ade7-31819dc76c46\" (UID: \"f206e5b3-c341-47ea-ade7-31819dc76c46\") "
Nov 02 15:37:35.209063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.198790    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca" pod "f206e5b3-c341-47ea-ade7-31819dc76c46" (UID: "f206e5b3-c341-47ea-ade7-31819dc76c46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:35.209945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.209913    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f206e5b3-c341-47ea-ade7-31819dc76c46-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:35.222314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.220861    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls" pod "f206e5b3-c341-47ea-ade7-31819dc76c46" (UID: "f206e5b3-c341-47ea-ade7-31819dc76c46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:35.223896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.223870    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs" pod "f206e5b3-c341-47ea-ade7-31819dc76c46" (UID: "f206e5b3-c341-47ea-ade7-31819dc76c46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:35.224907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.224766    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t" pod "f206e5b3-c341-47ea-ade7-31819dc76c46" (UID: "f206e5b3-c341-47ea-ade7-31819dc76c46"). InnerVolumeSpecName "kube-api-access-txr2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:35.281432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.281200    5041 generic.go:356] "Generic (PLEG): container finished" podID="f206e5b3-c341-47ea-ade7-31819dc76c46" containerID="b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9" exitCode=0
Nov 02 15:37:35.281611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.281455    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9" event={"ID":"f206e5b3-c341-47ea-ade7-31819dc76c46","Type":"ContainerDied","Data":"b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9"}
Nov 02 15:37:35.281611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.281499    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9" event={"ID":"f206e5b3-c341-47ea-ade7-31819dc76c46","Type":"ContainerDied","Data":"84a8105e669e2b19baa094050812fced2d8feebe645d04c43bae799937e5c415"}
Nov 02 15:37:35.281611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.281523    5041 scope.go:122] "RemoveContainer" containerID="b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9"
Nov 02 15:37:35.281797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.281655    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"
Nov 02 15:37:35.314559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.314524    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:35.314741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.314563    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txr2t\" (UniqueName: \"kubernetes.io/projected/f206e5b3-c341-47ea-ade7-31819dc76c46-kube-api-access-txr2t\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:35.314741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.314578    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f206e5b3-c341-47ea-ade7-31819dc76c46-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:35.320884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.320675    5041 scope.go:122] "RemoveContainer" containerID="b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9"
Nov 02 15:37:35.321049 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:35.321027    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9\": container with ID starting with b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9 not found: ID does not exist" containerID="b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9"
Nov 02 15:37:35.321103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.321058    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9"} err="failed to get container status \"b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9\": rpc error: code = NotFound desc = could not find container \"b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9\": container with ID starting with b6b8b507328e2733b1180504db3731218bd5a8b34e20bc3df35d1557d84cb3e9 not found: ID does not exist"
Nov 02 15:37:35.344438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.344394    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"]
Nov 02 15:37:35.352191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.352129    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69ff88-5f9c486fc7-r78d9"]
Nov 02 15:37:35.890279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:35.890247    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f206e5b3-c341-47ea-ade7-31819dc76c46" path="/var/lib/kubelet/pods/f206e5b3-c341-47ea-ade7-31819dc76c46/volumes"
Nov 02 15:37:36.441625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:36.441588    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:36.770100 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:36.770022    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:37:43.817396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:43.817327    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"]
Nov 02 15:37:43.817884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:43.817608    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw" podUID="7759d79f-bc30-4b32-9c66-a96be695caa4" containerName="maas-api" containerID="cri-o://2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef" gracePeriod=30
Nov 02 15:37:44.122487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.122455    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:37:44.261770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.261736    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls\") pod \"7759d79f-bc30-4b32-9c66-a96be695caa4\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") "
Nov 02 15:37:44.261974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.261788    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg\" (UniqueName: \"kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg\") pod \"7759d79f-bc30-4b32-9c66-a96be695caa4\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") "
Nov 02 15:37:44.261974 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.261948    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca\") pod \"7759d79f-bc30-4b32-9c66-a96be695caa4\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") "
Nov 02 15:37:44.262080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.262031    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs\") pod \"7759d79f-bc30-4b32-9c66-a96be695caa4\" (UID: \"7759d79f-bc30-4b32-9c66-a96be695caa4\") "
Nov 02 15:37:44.262341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.262306    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca" pod "7759d79f-bc30-4b32-9c66-a96be695caa4" (UID: "7759d79f-bc30-4b32-9c66-a96be695caa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:44.262736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.262709    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7759d79f-bc30-4b32-9c66-a96be695caa4-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:44.264100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.264072    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg" pod "7759d79f-bc30-4b32-9c66-a96be695caa4" (UID: "7759d79f-bc30-4b32-9c66-a96be695caa4"). InnerVolumeSpecName "kube-api-access-9xlhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:44.264100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.264081    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs" pod "7759d79f-bc30-4b32-9c66-a96be695caa4" (UID: "7759d79f-bc30-4b32-9c66-a96be695caa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:44.264230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.264157    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls" pod "7759d79f-bc30-4b32-9c66-a96be695caa4" (UID: "7759d79f-bc30-4b32-9c66-a96be695caa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:44.365572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.365469    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:44.365572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.365500    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlhg\" (UniqueName: \"kubernetes.io/projected/7759d79f-bc30-4b32-9c66-a96be695caa4-kube-api-access-9xlhg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:44.365572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.365510    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7759d79f-bc30-4b32-9c66-a96be695caa4-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:44.371150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.371111    5041 generic.go:356] "Generic (PLEG): container finished" podID="7759d79f-bc30-4b32-9c66-a96be695caa4" containerID="2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef" exitCode=0
Nov 02 15:37:44.371333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.371163    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"
Nov 02 15:37:44.371333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.371179    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw" event={"ID":"7759d79f-bc30-4b32-9c66-a96be695caa4","Type":"ContainerDied","Data":"2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef"}
Nov 02 15:37:44.371333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.371233    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw" event={"ID":"7759d79f-bc30-4b32-9c66-a96be695caa4","Type":"ContainerDied","Data":"7dfc8fa6206694445ed5e876554630880237ddcc2928c12a382f8bede07f5784"}
Nov 02 15:37:44.371333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.371263    5041 scope.go:122] "RemoveContainer" containerID="2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef"
Nov 02 15:37:44.384943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.384919    5041 scope.go:122] "RemoveContainer" containerID="2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef"
Nov 02 15:37:44.385234 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:37:44.385205    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef\": container with ID starting with 2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef not found: ID does not exist" containerID="2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef"
Nov 02 15:37:44.385299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.385236    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef"} err="failed to get container status \"2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef\": rpc error: code = NotFound desc = could not find container \"2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef\": container with ID starting with 2d5559378afd09b63ed9616df169f07573be7acda355580facbb932ea8089cef not found: ID does not exist"
Nov 02 15:37:44.430751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.430516    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"]
Nov 02 15:37:44.436131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:44.436105    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d35a86-8b876d49d-nmshw"]
Nov 02 15:37:45.890352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:45.890316    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7759d79f-bc30-4b32-9c66-a96be695caa4" path="/var/lib/kubelet/pods/7759d79f-bc30-4b32-9c66-a96be695caa4/volumes"
Nov 02 15:37:46.441100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:46.441063    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:47.573946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.573915    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"]
Nov 02 15:37:47.574669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574651    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerName="payload-processing"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574670    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerName="payload-processing"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574682    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574688    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574703    5041 cpu_manager.go:414] "containerMap: removing container" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerName="payload-pre-processing"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574709    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerName="payload-pre-processing"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574721    5041 cpu_manager.go:414] "containerMap: removing container" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" containerName="payload-pre-processing"
Nov 02 15:37:47.574727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574727    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" containerName="payload-pre-processing"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574735    5041 cpu_manager.go:414] "containerMap: removing container" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" containerName="payload-pre-processing"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574740    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" containerName="payload-pre-processing"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574754    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerName="payload-pre-processing"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574759    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerName="payload-pre-processing"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574769    5041 cpu_manager.go:414] "containerMap: removing container" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574773    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574785    5041 cpu_manager.go:414] "containerMap: removing container" podUID="55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574789    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574797    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7759d79f-bc30-4b32-9c66-a96be695caa4" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574802    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7759d79f-bc30-4b32-9c66-a96be695caa4" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574810    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574814    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574825    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f206e5b3-c341-47ea-ade7-31819dc76c46" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574829    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f206e5b3-c341-47ea-ade7-31819dc76c46" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574841    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574845    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574854    5041 cpu_manager.go:414] "containerMap: removing container" podUID="734bf2eb-db6f-4808-b21e-6d1bfb141987" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574859    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734bf2eb-db6f-4808-b21e-6d1bfb141987" containerName="maas-api"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574874    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574878    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574885    5041 cpu_manager.go:414] "containerMap: removing container" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerName="payload-processing"
Nov 02 15:37:47.574946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.574890    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerName="payload-processing"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575006    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbac19b-fed9-4bcd-a66d-ef9d24babefb" containerName="istio-proxy"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575021    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="738bb0ae-3436-4de8-9cbf-30a55e30a104" containerName="payload-pre-processing"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575034    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e4fb40-3326-425b-be8c-40c03199cf10" containerName="payload-pre-processing"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575046    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="75824f88-123f-4256-8369-ccb1efcbaa50" containerName="payload-pre-processing"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575058    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4bf93b2-7a33-4e17-9b79-53328a28761d" containerName="payload-pre-processing"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575067    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="417d8860-2269-40da-acb1-bccef5de32e9" containerName="istio-proxy"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575075    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ad83f6-7c13-4908-b2cb-9ffc34f39ce5" containerName="maas-api"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575081    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7759d79f-bc30-4b32-9c66-a96be695caa4" containerName="maas-api"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575088    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb17794-cb62-4a6e-bec6-45bc461b2a9b" containerName="istio-proxy"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575099    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f206e5b3-c341-47ea-ade7-31819dc76c46" containerName="maas-api"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575110    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98cf97b-261b-4c1b-bb6c-c5df51f9bd8e" containerName="maas-api"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575121    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="734bf2eb-db6f-4808-b21e-6d1bfb141987" containerName="maas-api"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575135    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8c355a-4867-4eee-98a4-0ccca21f6301" containerName="istio-proxy"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575142    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="687dc132-d1be-4d39-aa20-873e5a4191f0" containerName="payload-processing"
Nov 02 15:37:47.575755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.575148    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a621c27-1503-48ee-a17b-f25f9d4f0b3a" containerName="payload-processing"
Nov 02 15:37:47.580554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.580530    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.598663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.598644    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8a3fb0\"/\"openshift-service-ca.crt\""
Nov 02 15:37:47.600731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.600707    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.600846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.600745    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.600920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.600891    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.600981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.600927    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.600981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.600951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.601702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.601680    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvrt\" (UniqueName: \"kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.609915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.609892    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8a3fb0\"/\"kube-root-ca.crt\""
Nov 02 15:37:47.610465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.610445    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8a3fb0\"/\"default-dockercfg-rks4h\""
Nov 02 15:37:47.610564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.610465    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8a3fb0\"/\"auth-test-model-8a3fb0-kserve-self-signed-certs\""
Nov 02 15:37:47.615859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.615835    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"]
Nov 02 15:37:47.702948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.702909    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvrt\" (UniqueName: \"kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703075    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703116    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703156    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703182    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703740    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703811    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.703962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.703939    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.706065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.706041    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.706479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.706457    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.730768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.730738    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvrt\" (UniqueName: \"kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt\") pod \"auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") " pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:47.911867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.911794    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8a3fb0\"/\"default-dockercfg-rks4h\""
Nov 02 15:37:47.919346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:47.919313    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:48.235802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:48.235768    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"]
Nov 02 15:37:48.417813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:48.417773    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw" event={"ID":"d09f58ea-e7e2-4c1f-b506-27bbffd37f73","Type":"ContainerStarted","Data":"3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985"}
Nov 02 15:37:48.417985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:48.417824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw" event={"ID":"d09f58ea-e7e2-4c1f-b506-27bbffd37f73","Type":"ContainerStarted","Data":"40a0d8b5f5a017c637407aa949889eba76211e6c6f81941a1be9780a2921683a"}
Nov 02 15:37:48.463327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:48.463271    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw" podStartSLOduration=1.463257844 podStartE2EDuration="1.463257844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:48.459058229 +0000 UTC m=+2520.717127682" watchObservedRunningTime="2026-11-02 15:37:48.463257844 +0000 UTC m=+2520.721327294"
Nov 02 15:37:48.919622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:48.919590    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:52.453583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.453551    5041 generic.go:356] "Generic (PLEG): container finished" podID="11016326-7fb1-43f2-8377-007942c7fbe7" containerID="6e75610bcef7c1b867aa8e3d76b053448acebd593bece6fa98f316e350c0c868" exitCode=0
Nov 02 15:37:52.453932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.453625    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" event={"ID":"11016326-7fb1-43f2-8377-007942c7fbe7","Type":"ContainerDied","Data":"6e75610bcef7c1b867aa8e3d76b053448acebd593bece6fa98f316e350c0c868"}
Nov 02 15:37:52.453932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.453687    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69" event={"ID":"11016326-7fb1-43f2-8377-007942c7fbe7","Type":"ContainerDied","Data":"11ac1633adbcec866135eac777d0823842834ced761b7cd54dcc889de7fb170b"}
Nov 02 15:37:52.453932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.453710    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ac1633adbcec866135eac777d0823842834ced761b7cd54dcc889de7fb170b"
Nov 02 15:37:52.469353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.469332    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:37:52.568431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.568385    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg\" (UniqueName: \"kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg\") pod \"11016326-7fb1-43f2-8377-007942c7fbe7\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") "
Nov 02 15:37:52.568623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.568469    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume\") pod \"11016326-7fb1-43f2-8377-007942c7fbe7\" (UID: \"11016326-7fb1-43f2-8377-007942c7fbe7\") "
Nov 02 15:37:52.568925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.568900    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume" pod "11016326-7fb1-43f2-8377-007942c7fbe7" (UID: "11016326-7fb1-43f2-8377-007942c7fbe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:52.570692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.570663    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg" pod "11016326-7fb1-43f2-8377-007942c7fbe7" (UID: "11016326-7fb1-43f2-8377-007942c7fbe7"). InnerVolumeSpecName "kube-api-access-xfmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:52.670465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.670432    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11016326-7fb1-43f2-8377-007942c7fbe7-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:52.670465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:52.670462    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmvg\" (UniqueName: \"kubernetes.io/projected/11016326-7fb1-43f2-8377-007942c7fbe7-kube-api-access-xfmvg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:37:53.461203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:53.461164    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"
Nov 02 15:37:53.513504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:53.513476    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"]
Nov 02 15:37:53.524092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:53.524059    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69ff88-7c5b44f9ff-jvq69"]
Nov 02 15:37:53.889461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:53.889388    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" path="/var/lib/kubelet/pods/11016326-7fb1-43f2-8377-007942c7fbe7/volumes"
Nov 02 15:37:57.919912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:57.919878    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:57.932783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:57.932759    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:58.510872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.510840    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:37:58.897770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.897683    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"]
Nov 02 15:37:58.898828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.898802    5041 cpu_manager.go:414] "containerMap: removing container" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" containerName="payload-pre-processing"
Nov 02 15:37:58.898828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.898828    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" containerName="payload-pre-processing"
Nov 02 15:37:58.899059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.899047    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="11016326-7fb1-43f2-8377-007942c7fbe7" containerName="payload-pre-processing"
Nov 02 15:37:58.903892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.903871    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:58.927463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.927409    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6f2e0c\""
Nov 02 15:37:58.927874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.927852    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6f2e0c-dockercfg-gwq65\""
Nov 02 15:37:58.931505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.931483    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"]
Nov 02 15:37:58.957567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.957537    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:58.957715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:58.957626    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdmv\" (UniqueName: \"kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv\") pod \"payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:59.093514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:59.093478    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdmv\" (UniqueName: \"kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv\") pod \"payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:59.093849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:59.093793    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:59.094693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:59.094668    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:59.117041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:59.116985    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdmv\" (UniqueName: \"kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv\") pod \"payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:59.219241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:59.219207    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:37:59.541044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:37:59.541016    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"]
Nov 02 15:37:59.542517 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:37:59.542489    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046d2093_4395_4dc8_9a50_cdba9e799355.slice/crio-b7b071d3439efc0b4216c7dfd6b7c13b87b2d2ea8c28bea6e694a1618c4f8da8 WatchSource:0}: Error finding container b7b071d3439efc0b4216c7dfd6b7c13b87b2d2ea8c28bea6e694a1618c4f8da8: Status 404 returned error can't find the container with id b7b071d3439efc0b4216c7dfd6b7c13b87b2d2ea8c28bea6e694a1618c4f8da8
Nov 02 15:38:00.519545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:00.519503    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" event={"ID":"046d2093-4395-4dc8-9a50-cdba9e799355","Type":"ContainerStarted","Data":"52a0d19de5ce7d31f7e18bdacaf74eafe3b991335464ee16d1362cd6b10de1a5"}
Nov 02 15:38:00.520026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:00.519750    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:38:00.520026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:00.519771    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" event={"ID":"046d2093-4395-4dc8-9a50-cdba9e799355","Type":"ContainerStarted","Data":"b7b071d3439efc0b4216c7dfd6b7c13b87b2d2ea8c28bea6e694a1618c4f8da8"}
Nov 02 15:38:00.563272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:00.563203    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" podStartSLOduration=2.563185924 podStartE2EDuration="2.563185924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:00.561035912 +0000 UTC m=+2532.819105364" watchObservedRunningTime="2026-11-02 15:38:00.563185924 +0000 UTC m=+2532.821255405"
Nov 02 15:38:06.773324 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:38:06.773282    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:38:11.530317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:11.530281    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:38:36.770212 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:38:36.770127    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:38:37.241381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.239994    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"]
Nov 02 15:38:37.241381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.240377    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" podUID="b1b13603-aaac-454d-9734-807b6e69ffe7" containerName="discovery" containerID="cri-o://d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98" gracePeriod=30
Nov 02 15:38:37.548305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.548266    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:38:37.628970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.628930    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.629256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.629016    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.629256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.629052    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.629256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.629126    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.629256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.629177    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.629576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.629273    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.629576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.629507    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig\") pod \"b1b13603-aaac-454d-9734-807b6e69ffe7\" (UID: \"b1b13603-aaac-454d-9734-807b6e69ffe7\") "
Nov 02 15:38:37.630198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.630149    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:37.643995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.643892    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:38:37.643995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.643891    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:37.643995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.643923    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "kube-api-access-blll8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:37.643995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.643969    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:37.643995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.643970    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:38:37.644270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.643997    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts" pod "b1b13603-aaac-454d-9734-807b6e69ffe7" (UID: "b1b13603-aaac-454d-9734-807b6e69ffe7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:38:37.731395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731335    5041 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b13603-aaac-454d-9734-807b6e69ffe7-local-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.731548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731404    5041 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-cacerts\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.731548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731418    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.731548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731428    5041 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-ca-configmap\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.731548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731439    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blll8\" (UniqueName: \"kubernetes.io/projected/b1b13603-aaac-454d-9734-807b6e69ffe7-kube-api-access-blll8\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.731548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731448    5041 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-kubeconfig\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.731548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.731457    5041 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1b13603-aaac-454d-9734-807b6e69ffe7-istio-csr-dns-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:38:37.840002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.839935    5041 generic.go:356] "Generic (PLEG): container finished" podID="b1b13603-aaac-454d-9734-807b6e69ffe7" containerID="d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98" exitCode=0
Nov 02 15:38:37.840002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.839984    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"
Nov 02 15:38:37.840510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.839989    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" event={"ID":"b1b13603-aaac-454d-9734-807b6e69ffe7","Type":"ContainerDied","Data":"d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98"}
Nov 02 15:38:37.840510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.840106    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p" event={"ID":"b1b13603-aaac-454d-9734-807b6e69ffe7","Type":"ContainerDied","Data":"6c7d2d788bbd1a6cbdefd7e0b3c3b23c3ededcc16e9ade6708736ad3477ebff6"}
Nov 02 15:38:37.840510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.840139    5041 scope.go:122] "RemoveContainer" containerID="d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98"
Nov 02 15:38:37.863250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.863228    5041 scope.go:122] "RemoveContainer" containerID="d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98"
Nov 02 15:38:37.863535 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:38:37.863517    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98\": container with ID starting with d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98 not found: ID does not exist" containerID="d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98"
Nov 02 15:38:37.863591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.863543    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98"} err="failed to get container status \"d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98\": rpc error: code = NotFound desc = could not find container \"d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98\": container with ID starting with d044f804ecc9dc2d218dd84d5e8ff277b2156c97bdf93ff461a057af2ababe98 not found: ID does not exist"
Nov 02 15:38:37.897068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.897044    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"]
Nov 02 15:38:37.900857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:37.900835    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7t9p"]
Nov 02 15:38:39.338610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.338577    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"]
Nov 02 15:38:39.339686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.339658    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b1b13603-aaac-454d-9734-807b6e69ffe7" containerName="discovery"
Nov 02 15:38:39.339686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.339681    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b13603-aaac-454d-9734-807b6e69ffe7" containerName="discovery"
Nov 02 15:38:39.339907 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.339844    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b13603-aaac-454d-9734-807b6e69ffe7" containerName="discovery"
Nov 02 15:38:39.347608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.347585    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.371210 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.370957    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-453864\"/\"kube-root-ca.crt\""
Nov 02 15:38:39.373877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.373851    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"]
Nov 02 15:38:39.380288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.380253    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-453864\"/\"openshift-service-ca.crt\""
Nov 02 15:38:39.382629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.382607    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-453864\"/\"auth-test-model-453864-kserve-self-signed-certs\""
Nov 02 15:38:39.382746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.382707    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-453864\"/\"default-dockercfg-kf74s\""
Nov 02 15:38:39.450199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.450168    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.450305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.450220    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.450435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.450412    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7rp\" (UniqueName: \"kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.450474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.450461    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.450516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.450491    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.450596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.450580    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.551821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.551791    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.551933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.551845    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.551933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.551885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.552009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.551990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7rp\" (UniqueName: \"kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.552048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.552030    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.552083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.552070    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.552222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.552199    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.552267 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.552253    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.552437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.552419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.554190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.554164    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.554648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.554626    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.576039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.576011    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7rp\" (UniqueName: \"kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp\") pod \"auth-test-model-453864-kserve-5b9544ff57-kmlvm\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") " pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.662674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.662578    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:39.890718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.890681    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b13603-aaac-454d-9734-807b6e69ffe7" path="/var/lib/kubelet/pods/b1b13603-aaac-454d-9734-807b6e69ffe7/volumes"
Nov 02 15:38:39.976784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:39.976760    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"]
Nov 02 15:38:39.977310 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:38:39.977285    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78047769_ae49_463d_9d73_d35afa81b800.slice/crio-4768b750b44340498670fc6dd7a534783c9df608bb24f9732443bb7c1277c6a9 WatchSource:0}: Error finding container 4768b750b44340498670fc6dd7a534783c9df608bb24f9732443bb7c1277c6a9: Status 404 returned error can't find the container with id 4768b750b44340498670fc6dd7a534783c9df608bb24f9732443bb7c1277c6a9
Nov 02 15:38:40.866946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:40.866909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm" event={"ID":"78047769-ae49-463d-9d73-d35afa81b800","Type":"ContainerStarted","Data":"abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d"}
Nov 02 15:38:40.867352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:40.866962    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm" event={"ID":"78047769-ae49-463d-9d73-d35afa81b800","Type":"ContainerStarted","Data":"4768b750b44340498670fc6dd7a534783c9df608bb24f9732443bb7c1277c6a9"}
Nov 02 15:38:40.908597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:40.908544    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm" podStartSLOduration=1.90852765 podStartE2EDuration="1.90852765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:40.907601265 +0000 UTC m=+2573.165670736" watchObservedRunningTime="2026-11-02 15:38:40.90852765 +0000 UTC m=+2573.166597104"
Nov 02 15:38:41.662800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:41.662765    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:49.663150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:49.663117    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:49.675876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:49.675853    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:49.948844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:49.948813    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:38:50.001966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.001925    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"]
Nov 02 15:38:50.007032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.006976    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.044089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.043340    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0a7163\"/\"default-dockercfg-htlkn\""
Nov 02 15:38:50.044089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.043356    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0a7163\"/\"openshift-service-ca.crt\""
Nov 02 15:38:50.044089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.043349    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0a7163\"/\"kube-root-ca.crt\""
Nov 02 15:38:50.044089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.043464    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0a7163\"/\"rate-test-model-0a7163-kserve-self-signed-certs\""
Nov 02 15:38:50.046764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.046739    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"]
Nov 02 15:38:50.171750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.171709    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.171895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.171759    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.171895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.171862    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjzw\" (UniqueName: \"kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.171979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.171922    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.172016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.171999    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.172107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.172091    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.273983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.273894    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.273983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.273977    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274057    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274147    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274180    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274248    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjzw\" (UniqueName: \"kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274413    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.274984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.274964    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.277515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.277491    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.277799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.277765    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.298445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.298411    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjzw\" (UniqueName: \"kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw\") pod \"rate-test-model-0a7163-kserve-69b944f6c7-wwdhg\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") " pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.324707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.324676    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:50.653715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.653684    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"]
Nov 02 15:38:50.654139 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:38:50.654115    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1653e611_cfba_42ac_b836_e2564a999d77.slice/crio-6fc9c4acdd8c413fccc96969d4d573e4a084b1d465b287e2d7a50a71af2ec6b8 WatchSource:0}: Error finding container 6fc9c4acdd8c413fccc96969d4d573e4a084b1d465b287e2d7a50a71af2ec6b8: Status 404 returned error can't find the container with id 6fc9c4acdd8c413fccc96969d4d573e4a084b1d465b287e2d7a50a71af2ec6b8
Nov 02 15:38:50.948715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.948677    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg" event={"ID":"1653e611-cfba-42ac-b836-e2564a999d77","Type":"ContainerStarted","Data":"44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02"}
Nov 02 15:38:50.949217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.948724    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg" event={"ID":"1653e611-cfba-42ac-b836-e2564a999d77","Type":"ContainerStarted","Data":"6fc9c4acdd8c413fccc96969d4d573e4a084b1d465b287e2d7a50a71af2ec6b8"}
Nov 02 15:38:50.993177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:50.993126    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg" podStartSLOduration=1.9931136029999998 podStartE2EDuration="1.993113603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:50.990092792 +0000 UTC m=+2583.248162238" watchObservedRunningTime="2026-11-02 15:38:50.993113603 +0000 UTC m=+2583.251183052"
Nov 02 15:38:51.325214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:51.325136    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:38:51.659633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:51.659544    5041 scope.go:122] "RemoveContainer" containerID="6e75610bcef7c1b867aa8e3d76b053448acebd593bece6fa98f316e350c0c868"
Nov 02 15:38:59.806120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.806082    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"]
Nov 02 15:38:59.812290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.812267    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.842031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.842004    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f6f1a0-serving-cert\""
Nov 02 15:38:59.849547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.849520    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"]
Nov 02 15:38:59.885605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.885567    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.885762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.885710    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.885819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.885762    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmzh\" (UniqueName: \"kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.885865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.885850    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.987269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.987237    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.987478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.987311    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.987478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.987395    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzh\" (UniqueName: \"kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.987478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.987454    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.988139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.988112    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.989786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.989766    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:38:59.989896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:38:59.989787    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:39:00.010180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:00.010156    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmzh\" (UniqueName: \"kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh\") pod \"maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:39:00.135293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:00.135213    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:39:00.325455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:00.325414    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:39:00.338397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:00.338349    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:39:00.452869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:00.452835    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"]
Nov 02 15:39:00.453909 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:00.453881    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dd368d_0046_43ab_9fa6_0edbffa602c8.slice/crio-8d933fd909563089c2bd9e8011ab80cc90b4fb3d158f91ab69cc0eaa9d713146 WatchSource:0}: Error finding container 8d933fd909563089c2bd9e8011ab80cc90b4fb3d158f91ab69cc0eaa9d713146: Status 404 returned error can't find the container with id 8d933fd909563089c2bd9e8011ab80cc90b4fb3d158f91ab69cc0eaa9d713146
Nov 02 15:39:00.455587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:00.455564    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:39:01.037500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:01.037415    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2" event={"ID":"e4dd368d-0046-43ab-9fa6-0edbffa602c8","Type":"ContainerStarted","Data":"3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7"}
Nov 02 15:39:01.037500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:01.037476    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2" event={"ID":"e4dd368d-0046-43ab-9fa6-0edbffa602c8","Type":"ContainerStarted","Data":"8d933fd909563089c2bd9e8011ab80cc90b4fb3d158f91ab69cc0eaa9d713146"}
Nov 02 15:39:01.038008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:01.037511    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:39:01.050422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:01.050395    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:39:01.085676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:01.085626    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2" podStartSLOduration=1.75970272 podStartE2EDuration="2.085611735s" totalImagesPullingTime="325.909015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:01.083464847 +0000 UTC m=+2593.341534300" watchObservedRunningTime="2026-11-02 15:39:01.085611735 +0000 UTC m=+2593.343681186"
Nov 02 15:39:06.770007 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:39:06.769970    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:39:07.052852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:07.052783    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:39:15.462775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.462741    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"]
Nov 02 15:39:15.467142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.467124    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.485246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.485224    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-797488\""
Nov 02 15:39:15.495233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.495210    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-797488-dockercfg-z5vm4\""
Nov 02 15:39:15.500669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.500643    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"]
Nov 02 15:39:15.525153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.525127    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"]
Nov 02 15:39:15.530824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.530802    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.552322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.552294    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"]
Nov 02 15:39:15.589785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.589752    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqcmx\" (UniqueName: \"kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx\") pod \"payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.589951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.589931    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.691772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.691737    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.691942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.691845    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.692007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.691985    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdsk9\" (UniqueName: \"kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9\") pod \"payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.692112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.692088    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqcmx\" (UniqueName: \"kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx\") pod \"payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.692609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.692590    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.715731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.715669    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqcmx\" (UniqueName: \"kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx\") pod \"payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.782794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.782772    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:15.793982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.793952    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdsk9\" (UniqueName: \"kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9\") pod \"payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.794159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.794143    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.794861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.794835    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.817022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.816992    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdsk9\" (UniqueName: \"kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9\") pod \"payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:15.845071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:15.845048    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:16.088816 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:16.088772    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd365b9f3_ecd2_4eb2_a0d1_8b350288b8b3.slice/crio-059f23caabfbeaa210f8180cf2b000cce15a3bc1d07d15e3d8946151e5c7c02b WatchSource:0}: Error finding container 059f23caabfbeaa210f8180cf2b000cce15a3bc1d07d15e3d8946151e5c7c02b: Status 404 returned error can't find the container with id 059f23caabfbeaa210f8180cf2b000cce15a3bc1d07d15e3d8946151e5c7c02b
Nov 02 15:39:16.088932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:16.088888    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"]
Nov 02 15:39:16.164458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:16.164428    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" event={"ID":"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3","Type":"ContainerStarted","Data":"059f23caabfbeaa210f8180cf2b000cce15a3bc1d07d15e3d8946151e5c7c02b"}
Nov 02 15:39:16.173338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:16.173311    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"]
Nov 02 15:39:16.173510 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:16.173489    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3fa945_454d_4a0d_8584_13749fbc0e71.slice/crio-df7e6359f83b200957d35d30a64d48598175888fecb28b92045ada0841e3df24 WatchSource:0}: Error finding container df7e6359f83b200957d35d30a64d48598175888fecb28b92045ada0841e3df24: Status 404 returned error can't find the container with id df7e6359f83b200957d35d30a64d48598175888fecb28b92045ada0841e3df24
Nov 02 15:39:17.174227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.174187    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" event={"ID":"8d3fa945-454d-4a0d-8584-13749fbc0e71","Type":"ContainerStarted","Data":"b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348"}
Nov 02 15:39:17.174727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.174276    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" event={"ID":"8d3fa945-454d-4a0d-8584-13749fbc0e71","Type":"ContainerStarted","Data":"df7e6359f83b200957d35d30a64d48598175888fecb28b92045ada0841e3df24"}
Nov 02 15:39:17.174727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.174310    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:17.176092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.176062    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" event={"ID":"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3","Type":"ContainerStarted","Data":"400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1"}
Nov 02 15:39:17.176228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.176161    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:17.226878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.226826    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" podStartSLOduration=2.226812388 podStartE2EDuration="2.226812388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:17.223673282 +0000 UTC m=+2609.481742733" watchObservedRunningTime="2026-11-02 15:39:17.226812388 +0000 UTC m=+2609.484881837"
Nov 02 15:39:17.281243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:17.281185    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" podStartSLOduration=2.281168055 podStartE2EDuration="2.281168055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:17.276553972 +0000 UTC m=+2609.534623424" watchObservedRunningTime="2026-11-02 15:39:17.281168055 +0000 UTC m=+2609.539237508"
Nov 02 15:39:28.188170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:28.188142    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:39:28.188710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:28.188212    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:39:36.769485 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:39:36.769444    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:39:43.523038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:43.522996    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"]
Nov 02 15:39:47.344178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.344135    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"]
Nov 02 15:39:47.349255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.349236    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.373783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.373757    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-797488-serving-cert\""
Nov 02 15:39:47.380745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.380714    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"]
Nov 02 15:39:47.384036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.384008    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.384326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.384305    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.384753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.384733    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.384913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.384896    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2pcb\" (UniqueName: \"kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.486780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.486743    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.486780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.486782    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2pcb\" (UniqueName: \"kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.487039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.486807    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.487039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.486867    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.487039 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:39:47.486879    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-serving-cert: secret "maas-api-e2e-shared-b-w2-797488-serving-cert" not found
Nov 02 15:39:47.487039 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:39:47.486945    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls podName:2fcc6398-b336-4fc4-8a97-0add8ca0a52d nodeName:}" failed. No retries permitted until 2026-11-02 15:39:47.986931548 +0000 UTC m=+2640.245000975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" (UID: "2fcc6398-b336-4fc4-8a97-0add8ca0a52d") : secret "maas-api-e2e-shared-b-w2-797488-serving-cert" not found
Nov 02 15:39:47.487636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.487618    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.489358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.489339    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.514156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.514126    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2pcb\" (UniqueName: \"kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.994777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.994742    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:47.997182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:47.997153    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:48.271141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:48.271055    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:48.588943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:48.588911    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"]
Nov 02 15:39:48.589293 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:48.589267    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcc6398_b336_4fc4_8a97_0add8ca0a52d.slice/crio-403d1b9eed52660b91f2c163db1914694ee31e7d3b1be75c56fce0cefd3d9915 WatchSource:0}: Error finding container 403d1b9eed52660b91f2c163db1914694ee31e7d3b1be75c56fce0cefd3d9915: Status 404 returned error can't find the container with id 403d1b9eed52660b91f2c163db1914694ee31e7d3b1be75c56fce0cefd3d9915
Nov 02 15:39:49.466761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:49.466722    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" event={"ID":"2fcc6398-b336-4fc4-8a97-0add8ca0a52d","Type":"ContainerStarted","Data":"8449e770b0c7488ca896823bd5e15026597db2b3cdb0539581846b4af0def555"}
Nov 02 15:39:49.466944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:49.466783    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" event={"ID":"2fcc6398-b336-4fc4-8a97-0add8ca0a52d","Type":"ContainerStarted","Data":"403d1b9eed52660b91f2c163db1914694ee31e7d3b1be75c56fce0cefd3d9915"}
Nov 02 15:39:49.466944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:49.466919    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:49.528081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:49.528015    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" podStartSLOduration=2.208330911 podStartE2EDuration="2.528001276s" totalImagesPullingTime="319.670365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:49.526617943 +0000 UTC m=+2641.784687393" watchObservedRunningTime="2026-11-02 15:39:49.528001276 +0000 UTC m=+2641.786070726"
Nov 02 15:39:50.897477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.897443    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"]
Nov 02 15:39:50.902169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.902152    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:50.934265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.934229    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9cfd1f-dockercfg-xkn4d\""
Nov 02 15:39:50.934265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.934226    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9cfd1f\""
Nov 02 15:39:50.936466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.936433    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8n45\" (UniqueName: \"kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:50.936612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.936576    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:50.940408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:50.940384    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"]
Nov 02 15:39:51.002578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.002538    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"]
Nov 02 15:39:51.017777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.017745    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.030621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.030588    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"]
Nov 02 15:39:51.037966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.037930    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8n45\" (UniqueName: \"kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:51.042873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.042843    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.043201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.043177    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:51.044255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.044232    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8842r\" (UniqueName: \"kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r\") pod \"payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.044615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.044126    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:51.082499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.082434    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8n45\" (UniqueName: \"kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:51.145422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.145346    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8842r\" (UniqueName: \"kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r\") pod \"payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.145624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.145575    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.146254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.146233    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.168760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.168733    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8842r\" (UniqueName: \"kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r\") pod \"payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.217519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.217493    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:51.351878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.351844    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:51.532757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.532726    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"]
Nov 02 15:39:51.532886 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:51.532786    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126d201a_bddd_4696_98db_41127cdea356.slice/crio-6708b55606279d929ce8cdcc7a1a5ad3b2a2ac292b9d73d1c8fae32412389930 WatchSource:0}: Error finding container 6708b55606279d929ce8cdcc7a1a5ad3b2a2ac292b9d73d1c8fae32412389930: Status 404 returned error can't find the container with id 6708b55606279d929ce8cdcc7a1a5ad3b2a2ac292b9d73d1c8fae32412389930
Nov 02 15:39:51.664174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:51.663135    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"]
Nov 02 15:39:51.665996 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:51.665971    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8ae5e8_fb0d_4863_b39a_285061b4978c.slice/crio-f9f1d48fbe802961e0d7668a42c2159a8a3ac137ce2f038bbec3b2897bf4c742 WatchSource:0}: Error finding container f9f1d48fbe802961e0d7668a42c2159a8a3ac137ce2f038bbec3b2897bf4c742: Status 404 returned error can't find the container with id f9f1d48fbe802961e0d7668a42c2159a8a3ac137ce2f038bbec3b2897bf4c742
Nov 02 15:39:52.498114 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.498077    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8" event={"ID":"126d201a-bddd-4696-98db-41127cdea356","Type":"ContainerStarted","Data":"980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23"}
Nov 02 15:39:52.498574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.498142    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:39:52.498574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.498170    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8" event={"ID":"126d201a-bddd-4696-98db-41127cdea356","Type":"ContainerStarted","Data":"6708b55606279d929ce8cdcc7a1a5ad3b2a2ac292b9d73d1c8fae32412389930"}
Nov 02 15:39:52.499894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.499862    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk" event={"ID":"3b8ae5e8-fb0d-4863-b39a-285061b4978c","Type":"ContainerStarted","Data":"0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27"}
Nov 02 15:39:52.500018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.499914    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk" event={"ID":"3b8ae5e8-fb0d-4863-b39a-285061b4978c","Type":"ContainerStarted","Data":"f9f1d48fbe802961e0d7668a42c2159a8a3ac137ce2f038bbec3b2897bf4c742"}
Nov 02 15:39:52.500018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.499940    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:39:52.543465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.543421    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8" podStartSLOduration=2.543406832 podStartE2EDuration="2.543406832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:52.537982437 +0000 UTC m=+2644.796051886" watchObservedRunningTime="2026-11-02 15:39:52.543406832 +0000 UTC m=+2644.801476632"
Nov 02 15:39:52.580687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:52.580642    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk" podStartSLOduration=2.5806295219999997 podStartE2EDuration="2.580629522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:52.576946594 +0000 UTC m=+2644.835016042" watchObservedRunningTime="2026-11-02 15:39:52.580629522 +0000 UTC m=+2644.838699020"
Nov 02 15:39:55.481324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:55.481290    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:39:56.361763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.361728    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"]
Nov 02 15:39:56.366690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.366672    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.398441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.398419    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-28da58\""
Nov 02 15:39:56.413477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.413448    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-28da58-dockercfg-26ck8\""
Nov 02 15:39:56.423513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.417097    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6w5\" (UniqueName: \"kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5\") pod \"payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.423513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.417153    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.423513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.419882    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"]
Nov 02 15:39:56.490252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.490198    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"]
Nov 02 15:39:56.495672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.495644    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.518515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.518472    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6w5\" (UniqueName: \"kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5\") pod \"payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.518686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.518530    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.518686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.518603    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgx59\" (UniqueName: \"kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59\") pod \"payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.518823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.518760    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.519492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.519463    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.545880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.545844    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"]
Nov 02 15:39:56.547976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.547946    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6w5\" (UniqueName: \"kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5\") pod \"payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.620831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.620758    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx59\" (UniqueName: \"kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59\") pod \"payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.620971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.620914    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.621850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.621825    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.643544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.643515    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx59\" (UniqueName: \"kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59\") pod \"payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.681911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.681885    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:56.819881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.819764    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:56.996737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:56.996704    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"]
Nov 02 15:39:56.996889 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:56.996708    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3fdefc_b125_4e26_ac80_5721ea18be6c.slice/crio-174ada3425241529f4dc2bfcac5e97a5890f76058155bde69b48f970c3a86d73 WatchSource:0}: Error finding container 174ada3425241529f4dc2bfcac5e97a5890f76058155bde69b48f970c3a86d73: Status 404 returned error can't find the container with id 174ada3425241529f4dc2bfcac5e97a5890f76058155bde69b48f970c3a86d73
Nov 02 15:39:57.149349 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:39:57.149285    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e68191_fd7d_400a_8a9c_3e6bbc242c81.slice/crio-1fa751c109f2f654b31437819bcdc7fca9af921e28b20579ac90bbaada1c1372 WatchSource:0}: Error finding container 1fa751c109f2f654b31437819bcdc7fca9af921e28b20579ac90bbaada1c1372: Status 404 returned error can't find the container with id 1fa751c109f2f654b31437819bcdc7fca9af921e28b20579ac90bbaada1c1372
Nov 02 15:39:57.151622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.151597    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"]
Nov 02 15:39:57.544239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.544204    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" event={"ID":"e9e68191-fd7d-400a-8a9c-3e6bbc242c81","Type":"ContainerStarted","Data":"02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00"}
Nov 02 15:39:57.544711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.544298    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" event={"ID":"e9e68191-fd7d-400a-8a9c-3e6bbc242c81","Type":"ContainerStarted","Data":"1fa751c109f2f654b31437819bcdc7fca9af921e28b20579ac90bbaada1c1372"}
Nov 02 15:39:57.544711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.544329    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:39:57.546103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.546074    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" event={"ID":"0f3fdefc-b125-4e26-ac80-5721ea18be6c","Type":"ContainerStarted","Data":"8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1"}
Nov 02 15:39:57.546227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.546118    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" event={"ID":"0f3fdefc-b125-4e26-ac80-5721ea18be6c","Type":"ContainerStarted","Data":"174ada3425241529f4dc2bfcac5e97a5890f76058155bde69b48f970c3a86d73"}
Nov 02 15:39:57.546227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.546157    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:39:57.590836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:57.590799    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" podStartSLOduration=1.590787038 podStartE2EDuration="1.590787038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:57.588185694 +0000 UTC m=+2649.846255145" watchObservedRunningTime="2026-11-02 15:39:57.590787038 +0000 UTC m=+2649.848856507"
Nov 02 15:39:58.553863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.553819    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5" podUID="e6e8d9ea-1522-4da9-9f84-513fc2281fee" containerName="main" containerID="cri-o://01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923" gracePeriod=45
Nov 02 15:39:58.858864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.858838    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:39:58.900857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.900805    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" podStartSLOduration=2.900791338 podStartE2EDuration="2.900791338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:57.6352681 +0000 UTC m=+2649.893337549" watchObservedRunningTime="2026-11-02 15:39:58.900791338 +0000 UTC m=+2651.158860791"
Nov 02 15:39:58.953067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj\" (UniqueName: \"kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj\") pod \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") "
Nov 02 15:39:58.953200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953079    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs\") pod \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") "
Nov 02 15:39:58.953200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953139    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache\") pod \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") "
Nov 02 15:39:58.953331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953203    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir\") pod \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") "
Nov 02 15:39:58.953331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953264    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm\") pod \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") "
Nov 02 15:39:58.953331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953297    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home\") pod \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\" (UID: \"e6e8d9ea-1522-4da9-9f84-513fc2281fee\") "
Nov 02 15:39:58.953514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953484    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache" pod "e6e8d9ea-1522-4da9-9f84-513fc2281fee" (UID: "e6e8d9ea-1522-4da9-9f84-513fc2281fee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:58.953611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953587    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir" pod "e6e8d9ea-1522-4da9-9f84-513fc2281fee" (UID: "e6e8d9ea-1522-4da9-9f84-513fc2281fee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:58.953664 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.953626    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home" pod "e6e8d9ea-1522-4da9-9f84-513fc2281fee" (UID: "e6e8d9ea-1522-4da9-9f84-513fc2281fee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:58.954166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.954142    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:39:58.954258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.954171    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:39:58.954258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.954186    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:39:58.955585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.955565    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs" pod "e6e8d9ea-1522-4da9-9f84-513fc2281fee" (UID: "e6e8d9ea-1522-4da9-9f84-513fc2281fee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:58.955687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.955580    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm" pod "e6e8d9ea-1522-4da9-9f84-513fc2281fee" (UID: "e6e8d9ea-1522-4da9-9f84-513fc2281fee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:58.955799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:58.955778    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj" pod "e6e8d9ea-1522-4da9-9f84-513fc2281fee" (UID: "e6e8d9ea-1522-4da9-9f84-513fc2281fee"). InnerVolumeSpecName "kube-api-access-2gqsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:59.055352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.055323    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqsj\" (UniqueName: \"kubernetes.io/projected/e6e8d9ea-1522-4da9-9f84-513fc2281fee-kube-api-access-2gqsj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:39:59.055352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.055350    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e8d9ea-1522-4da9-9f84-513fc2281fee-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:39:59.055518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.055383    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e8d9ea-1522-4da9-9f84-513fc2281fee-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:39:59.564635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.564604    5041 generic.go:356] "Generic (PLEG): container finished" podID="e6e8d9ea-1522-4da9-9f84-513fc2281fee" containerID="01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923" exitCode=0
Nov 02 15:39:59.564635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.564638    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5" event={"ID":"e6e8d9ea-1522-4da9-9f84-513fc2281fee","Type":"ContainerDied","Data":"01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923"}
Nov 02 15:39:59.565278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.564660    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"
Nov 02 15:39:59.565278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.564664    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5" event={"ID":"e6e8d9ea-1522-4da9-9f84-513fc2281fee","Type":"ContainerDied","Data":"ba2b2aa7fb61e61b3baebe4740d530feeaf6f79e99fa6460c2573692dca9b06b"}
Nov 02 15:39:59.565278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.564674    5041 scope.go:122] "RemoveContainer" containerID="01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923"
Nov 02 15:39:59.577253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.577227    5041 scope.go:122] "RemoveContainer" containerID="01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923"
Nov 02 15:39:59.577541 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:39:59.577520    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923\": container with ID starting with 01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923 not found: ID does not exist" containerID="01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923"
Nov 02 15:39:59.577607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.577547    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923"} err="failed to get container status \"01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923\": rpc error: code = NotFound desc = could not find container \"01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923\": container with ID starting with 01db2919d69ac29a169a779ae329b950dc693ba2340e461f26dd6a800432b923 not found: ID does not exist"
Nov 02 15:39:59.618395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.618357    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"]
Nov 02 15:39:59.628982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.628957    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-78d075/auth-test-model-78d075-kserve-666f9469bf-nscv5"]
Nov 02 15:39:59.889703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:39:59.889631    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e8d9ea-1522-4da9-9f84-513fc2281fee" path="/var/lib/kubelet/pods/e6e8d9ea-1522-4da9-9f84-513fc2281fee/volumes"
Nov 02 15:40:03.511992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:03.511956    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:40:03.512414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:03.512044    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:40:06.769706 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:40:06.769669    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:40:08.558404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:08.558343    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:40:08.558839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:08.558570    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:40:16.238899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:16.238864    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"]
Nov 02 15:40:23.206142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.206098    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"]
Nov 02 15:40:23.207161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.207134    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e6e8d9ea-1522-4da9-9f84-513fc2281fee" containerName="main"
Nov 02 15:40:23.207161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.207160    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e8d9ea-1522-4da9-9f84-513fc2281fee" containerName="main"
Nov 02 15:40:23.207425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.207408    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e8d9ea-1522-4da9-9f84-513fc2281fee" containerName="main"
Nov 02 15:40:23.211488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.211469    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.228433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.227627    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9cfd1f-serving-cert\""
Nov 02 15:40:23.256486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.256455    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"]
Nov 02 15:40:23.354828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.354791    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.354986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.354836    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.355126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.355103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.355166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.355137    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjkm\" (UniqueName: \"kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.457061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.456959    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.457061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.457000    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjkm\" (UniqueName: \"kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.457061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.457047    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.457452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.457071    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.457811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.457782    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.459677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.459652    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.459677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.459654    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.479983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.479955    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjkm\" (UniqueName: \"kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm\") pod \"maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.531162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.531122    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:23.841753 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:40:23.841722    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4174a84d_f201_40ff_8310_dfc750701f87.slice/crio-5832b27895bf5fc0aa0a1444d4f508321e5d76d7b182efece279d3b60337fa09 WatchSource:0}: Error finding container 5832b27895bf5fc0aa0a1444d4f508321e5d76d7b182efece279d3b60337fa09: Status 404 returned error can't find the container with id 5832b27895bf5fc0aa0a1444d4f508321e5d76d7b182efece279d3b60337fa09
Nov 02 15:40:23.842911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:23.842885    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"]
Nov 02 15:40:24.781443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:24.781403    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" event={"ID":"4174a84d-f201-40ff-8310-dfc750701f87","Type":"ContainerStarted","Data":"a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51"}
Nov 02 15:40:24.781850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:24.781486    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:24.781850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:24.781504    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" event={"ID":"4174a84d-f201-40ff-8310-dfc750701f87","Type":"ContainerStarted","Data":"5832b27895bf5fc0aa0a1444d4f508321e5d76d7b182efece279d3b60337fa09"}
Nov 02 15:40:24.833815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:24.833767    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" podStartSLOduration=1.5524316009999999 podStartE2EDuration="1.833750587s" totalImagesPullingTime="281.318986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:24.830084736 +0000 UTC m=+2677.088154187" watchObservedRunningTime="2026-11-02 15:40:24.833750587 +0000 UTC m=+2677.091820036"
Nov 02 15:40:30.796981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:30.796946    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:40:31.260802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.260766    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb" podUID="c44cb54a-3dfe-40e1-b210-5633bd692a7f" containerName="main" containerID="cri-o://1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0" gracePeriod=45
Nov 02 15:40:31.546528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.546500    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:40:31.637942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.637905    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"]
Nov 02 15:40:31.638804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.638785    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c44cb54a-3dfe-40e1-b210-5633bd692a7f" containerName="main"
Nov 02 15:40:31.638864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.638806    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44cb54a-3dfe-40e1-b210-5633bd692a7f" containerName="main"
Nov 02 15:40:31.638954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.638944    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44cb54a-3dfe-40e1-b210-5633bd692a7f" containerName="main"
Nov 02 15:40:31.642746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.642731    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.653865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.653840    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4\" (UniqueName: \"kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4\") pod \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") "
Nov 02 15:40:31.654001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.653895    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm\") pod \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") "
Nov 02 15:40:31.654077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654000    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs\") pod \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") "
Nov 02 15:40:31.654077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654070    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home\") pod \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") "
Nov 02 15:40:31.654195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654121    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir\") pod \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") "
Nov 02 15:40:31.654305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654286    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache\") pod \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\" (UID: \"c44cb54a-3dfe-40e1-b210-5633bd692a7f\") "
Nov 02 15:40:31.654408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654381    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home" pod "c44cb54a-3dfe-40e1-b210-5633bd692a7f" (UID: "c44cb54a-3dfe-40e1-b210-5633bd692a7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:31.654508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654466    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir" pod "c44cb54a-3dfe-40e1-b210-5633bd692a7f" (UID: "c44cb54a-3dfe-40e1-b210-5633bd692a7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:31.654685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654660    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache" pod "c44cb54a-3dfe-40e1-b210-5633bd692a7f" (UID: "c44cb54a-3dfe-40e1-b210-5633bd692a7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:31.654989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654966    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:40:31.655063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.654996    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:40:31.655063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.655013    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:40:31.656309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.656285    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm" pod "c44cb54a-3dfe-40e1-b210-5633bd692a7f" (UID: "c44cb54a-3dfe-40e1-b210-5633bd692a7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:31.656416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.656315    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4" pod "c44cb54a-3dfe-40e1-b210-5633bd692a7f" (UID: "c44cb54a-3dfe-40e1-b210-5633bd692a7f"). InnerVolumeSpecName "kube-api-access-p74f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:31.656416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.656323    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs" pod "c44cb54a-3dfe-40e1-b210-5633bd692a7f" (UID: "c44cb54a-3dfe-40e1-b210-5633bd692a7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:31.661737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.661713    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-28da58-serving-cert\""
Nov 02 15:40:31.669674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.669647    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"]
Nov 02 15:40:31.756239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756210    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.756419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756254    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.756419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756336    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.756545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756502    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgzh\" (UniqueName: \"kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.756648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756632    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74f4\" (UniqueName: \"kubernetes.io/projected/c44cb54a-3dfe-40e1-b210-5633bd692a7f-kube-api-access-p74f4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:40:31.756707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756650    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cb54a-3dfe-40e1-b210-5633bd692a7f-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:40:31.756707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.756661    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cb54a-3dfe-40e1-b210-5633bd692a7f-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:40:31.847311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.847200    5041 generic.go:356] "Generic (PLEG): container finished" podID="c44cb54a-3dfe-40e1-b210-5633bd692a7f" containerID="1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0" exitCode=0
Nov 02 15:40:31.847311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.847250    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"
Nov 02 15:40:31.847311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.847287    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb" event={"ID":"c44cb54a-3dfe-40e1-b210-5633bd692a7f","Type":"ContainerDied","Data":"1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0"}
Nov 02 15:40:31.847916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.847339    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb" event={"ID":"c44cb54a-3dfe-40e1-b210-5633bd692a7f","Type":"ContainerDied","Data":"9db8fc7bc53811c4ff752ae2f52abec84cdcd6d8fbc9b9470c1e6771eeb96484"}
Nov 02 15:40:31.847916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.847358    5041 scope.go:122] "RemoveContainer" containerID="1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0"
Nov 02 15:40:31.857427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.857403    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.857547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.857465    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.857547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.857500    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.857663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.857649    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgzh\" (UniqueName: \"kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.858292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.858267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.860060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.860037    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.860147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.860088    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.860263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.860244    5041 scope.go:122] "RemoveContainer" containerID="1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0"
Nov 02 15:40:31.860506 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:40:31.860488    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0\": container with ID starting with 1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0 not found: ID does not exist" containerID="1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0"
Nov 02 15:40:31.860557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.860512    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0"} err="failed to get container status \"1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0\": rpc error: code = NotFound desc = could not find container \"1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0\": container with ID starting with 1430a36dd9157bda016712cd85df62542d1c115477ab11cbc510224fea7bd9b0 not found: ID does not exist"
Nov 02 15:40:31.883017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.882994    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgzh\" (UniqueName: \"kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh\") pod \"maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:31.899909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.899862    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"]
Nov 02 15:40:31.904173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.904147    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/auth-test-model-cce791-kserve-6fc997ccdf-r56cb"]
Nov 02 15:40:31.958217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:31.958195    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:32.278387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:32.278330    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"]
Nov 02 15:40:32.280089 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:40:32.280063    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027e6fd4_3b11_45ce_a79e_6a08849f7005.slice/crio-9d4dbe686e361912f81e2d5df37dddd25858c301c53947b6c27c4b6f55b2200b WatchSource:0}: Error finding container 9d4dbe686e361912f81e2d5df37dddd25858c301c53947b6c27c4b6f55b2200b: Status 404 returned error can't find the container with id 9d4dbe686e361912f81e2d5df37dddd25858c301c53947b6c27c4b6f55b2200b
Nov 02 15:40:32.859522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:32.859164    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t" event={"ID":"027e6fd4-3b11-45ce-a79e-6a08849f7005","Type":"ContainerStarted","Data":"b9b86dbe00ad6f3c50ca7653fc7da88ba221ce952eca52b64a719cb45e3eabae"}
Nov 02 15:40:32.859522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:32.859239    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:32.859522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:32.859267    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t" event={"ID":"027e6fd4-3b11-45ce-a79e-6a08849f7005","Type":"ContainerStarted","Data":"9d4dbe686e361912f81e2d5df37dddd25858c301c53947b6c27c4b6f55b2200b"}
Nov 02 15:40:32.909181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:32.909138    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t" podStartSLOduration=1.505095098 podStartE2EDuration="1.909124972s" totalImagesPullingTime="404.029874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:32.9078668 +0000 UTC m=+2685.165936252" watchObservedRunningTime="2026-11-02 15:40:32.909124972 +0000 UTC m=+2685.167194415"
Nov 02 15:40:33.889945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:33.889908    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44cb54a-3dfe-40e1-b210-5633bd692a7f" path="/var/lib/kubelet/pods/c44cb54a-3dfe-40e1-b210-5633bd692a7f/volumes"
Nov 02 15:40:36.771592 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:40:36.771550    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:40:38.876032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:38.876001    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:40:49.181223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:49.181190    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:40:49.198941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:40:49.198915    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:41:05.471172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:05.471134    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"]
Nov 02 15:41:05.514914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:05.514877    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"]
Nov 02 15:41:06.770048 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:41:06.770003    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:41:10.472143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.472100    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk" podUID="3b8ae5e8-fb0d-4863-b39a-285061b4978c" containerName="payload-processing" containerID="cri-o://0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27" gracePeriod=25
Nov 02 15:41:10.515300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.515264    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8" podUID="126d201a-bddd-4696-98db-41127cdea356" containerName="payload-pre-processing" containerID="cri-o://980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23" gracePeriod=25
Nov 02 15:41:10.826231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.826199    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:41:10.857388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.857351    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:41:10.929751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.929721    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume\") pod \"126d201a-bddd-4696-98db-41127cdea356\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") "
Nov 02 15:41:10.930063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.929790    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume\") pod \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") "
Nov 02 15:41:10.930063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.930015    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r\" (UniqueName: \"kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r\") pod \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\" (UID: \"3b8ae5e8-fb0d-4863-b39a-285061b4978c\") "
Nov 02 15:41:10.930199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.930095    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45\" (UniqueName: \"kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45\") pod \"126d201a-bddd-4696-98db-41127cdea356\" (UID: \"126d201a-bddd-4696-98db-41127cdea356\") "
Nov 02 15:41:10.930199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.930160    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume" pod "126d201a-bddd-4696-98db-41127cdea356" (UID: "126d201a-bddd-4696-98db-41127cdea356"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:41:10.930314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.930237    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume" pod "3b8ae5e8-fb0d-4863-b39a-285061b4978c" (UID: "3b8ae5e8-fb0d-4863-b39a-285061b4978c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:41:10.930818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.930769    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126d201a-bddd-4696-98db-41127cdea356-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:41:10.930818 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.930797    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8ae5e8-fb0d-4863-b39a-285061b4978c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:41:10.932863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.932833    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r" pod "3b8ae5e8-fb0d-4863-b39a-285061b4978c" (UID: "3b8ae5e8-fb0d-4863-b39a-285061b4978c"). InnerVolumeSpecName "kube-api-access-8842r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:10.933032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:10.933009    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45" pod "126d201a-bddd-4696-98db-41127cdea356" (UID: "126d201a-bddd-4696-98db-41127cdea356"). InnerVolumeSpecName "kube-api-access-c8n45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:11.033272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.033195    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8842r\" (UniqueName: \"kubernetes.io/projected/3b8ae5e8-fb0d-4863-b39a-285061b4978c-kube-api-access-8842r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:41:11.033272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.033229    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8n45\" (UniqueName: \"kubernetes.io/projected/126d201a-bddd-4696-98db-41127cdea356-kube-api-access-c8n45\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:41:11.193775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.193738    5041 generic.go:356] "Generic (PLEG): container finished" podID="126d201a-bddd-4696-98db-41127cdea356" containerID="980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23" exitCode=0
Nov 02 15:41:11.193973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.193794    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"
Nov 02 15:41:11.193973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.193830    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8" event={"ID":"126d201a-bddd-4696-98db-41127cdea356","Type":"ContainerDied","Data":"980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23"}
Nov 02 15:41:11.193973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.193898    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8" event={"ID":"126d201a-bddd-4696-98db-41127cdea356","Type":"ContainerDied","Data":"6708b55606279d929ce8cdcc7a1a5ad3b2a2ac292b9d73d1c8fae32412389930"}
Nov 02 15:41:11.193973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.193929    5041 scope.go:122] "RemoveContainer" containerID="980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23"
Nov 02 15:41:11.195319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.195297    5041 generic.go:356] "Generic (PLEG): container finished" podID="3b8ae5e8-fb0d-4863-b39a-285061b4978c" containerID="0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27" exitCode=0
Nov 02 15:41:11.195449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.195340    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"
Nov 02 15:41:11.195449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.195416    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk" event={"ID":"3b8ae5e8-fb0d-4863-b39a-285061b4978c","Type":"ContainerDied","Data":"0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27"}
Nov 02 15:41:11.195606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.195458    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk" event={"ID":"3b8ae5e8-fb0d-4863-b39a-285061b4978c","Type":"ContainerDied","Data":"f9f1d48fbe802961e0d7668a42c2159a8a3ac137ce2f038bbec3b2897bf4c742"}
Nov 02 15:41:11.207521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.207496    5041 scope.go:122] "RemoveContainer" containerID="980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23"
Nov 02 15:41:11.207911 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:41:11.207876    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23\": container with ID starting with 980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23 not found: ID does not exist" containerID="980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23"
Nov 02 15:41:11.208027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.207936    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23"} err="failed to get container status \"980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23\": rpc error: code = NotFound desc = could not find container \"980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23\": container with ID starting with 980f1b3f6821ed51c07afeeecbd1d620085ff49fc76fb189c2b4eb7934129a23 not found: ID does not exist"
Nov 02 15:41:11.208027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.207959    5041 scope.go:122] "RemoveContainer" containerID="0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27"
Nov 02 15:41:11.218901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.218884    5041 scope.go:122] "RemoveContainer" containerID="0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27"
Nov 02 15:41:11.219163 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:41:11.219141    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27\": container with ID starting with 0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27 not found: ID does not exist" containerID="0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27"
Nov 02 15:41:11.219227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.219173    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27"} err="failed to get container status \"0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27\": rpc error: code = NotFound desc = could not find container \"0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27\": container with ID starting with 0200f09f0662d84b6cdda11c41939e68550b10ea8359c5ad4bc6319c3cb09b27 not found: ID does not exist"
Nov 02 15:41:11.245744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.245718    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"]
Nov 02 15:41:11.250020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.249994    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-6b76bcdbcf-mm5m8"]
Nov 02 15:41:11.285436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.285381    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"]
Nov 02 15:41:11.289390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.289355    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-54c6ff7bbc-vq8pk"]
Nov 02 15:41:11.890449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.890414    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126d201a-bddd-4696-98db-41127cdea356" path="/var/lib/kubelet/pods/126d201a-bddd-4696-98db-41127cdea356/volumes"
Nov 02 15:41:11.890891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:11.890873    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8ae5e8-fb0d-4863-b39a-285061b4978c" path="/var/lib/kubelet/pods/3b8ae5e8-fb0d-4863-b39a-285061b4978c/volumes"
Nov 02 15:41:16.703920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.703883    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"]
Nov 02 15:41:16.704705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.704686    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3b8ae5e8-fb0d-4863-b39a-285061b4978c" containerName="payload-processing"
Nov 02 15:41:16.704753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.704707    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8ae5e8-fb0d-4863-b39a-285061b4978c" containerName="payload-processing"
Nov 02 15:41:16.704753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.704729    5041 cpu_manager.go:414] "containerMap: removing container" podUID="126d201a-bddd-4696-98db-41127cdea356" containerName="payload-pre-processing"
Nov 02 15:41:16.704753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.704734    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126d201a-bddd-4696-98db-41127cdea356" containerName="payload-pre-processing"
Nov 02 15:41:16.704906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.704896    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8ae5e8-fb0d-4863-b39a-285061b4978c" containerName="payload-processing"
Nov 02 15:41:16.704946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.704911    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="126d201a-bddd-4696-98db-41127cdea356" containerName="payload-pre-processing"
Nov 02 15:41:16.708852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.708834    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.726220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.726193    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"openshift-service-ca.crt\""
Nov 02 15:41:16.737412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.737389    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"kube-root-ca.crt\""
Nov 02 15:41:16.741982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.741961    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"]
Nov 02 15:41:16.743013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.742990    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"sub-test-model-cce791-kserve-self-signed-certs\""
Nov 02 15:41:16.743545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.743529    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce791\"/\"default-dockercfg-tbgvn\""
Nov 02 15:41:16.802567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.802539    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.802678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.802587    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.802678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.802605    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.802763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.802722    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.802800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.802765    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pns\" (UniqueName: \"kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.802883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.802864    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.904498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.904463    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.904498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.904500    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.904728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.904554    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.904728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.904585    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pns\" (UniqueName: \"kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.904728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.904628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.904914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.904783    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.905045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.905014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.905173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.905132    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.905270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.905250    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.907242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.907214    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.907436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.907306    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:16.927792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:16.927754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pns\" (UniqueName: \"kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns\") pod \"sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") " pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:17.021830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:17.021738    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:17.358141 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:41:17.358106    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2055d9e5_cd3f_4fcb_8ba9_ba5439793983.slice/crio-17c5a01c0578d8a71f9e9a4bcdaee36f0b473b8d8e80c1bda2a6c9a18d3faf23 WatchSource:0}: Error finding container 17c5a01c0578d8a71f9e9a4bcdaee36f0b473b8d8e80c1bda2a6c9a18d3faf23: Status 404 returned error can't find the container with id 17c5a01c0578d8a71f9e9a4bcdaee36f0b473b8d8e80c1bda2a6c9a18d3faf23
Nov 02 15:41:17.361943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:17.361914    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"]
Nov 02 15:41:18.258621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:18.258582    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh" event={"ID":"2055d9e5-cd3f-4fcb-8ba9-ba5439793983","Type":"ContainerStarted","Data":"44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4"}
Nov 02 15:41:18.259010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:18.258633    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh" event={"ID":"2055d9e5-cd3f-4fcb-8ba9-ba5439793983","Type":"ContainerStarted","Data":"17c5a01c0578d8a71f9e9a4bcdaee36f0b473b8d8e80c1bda2a6c9a18d3faf23"}
Nov 02 15:41:18.306283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:18.306229    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh" podStartSLOduration=2.306211313 podStartE2EDuration="2.306211313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:18.30099668 +0000 UTC m=+2730.559066130" watchObservedRunningTime="2026-11-02 15:41:18.306211313 +0000 UTC m=+2730.564280804"
Nov 02 15:41:19.022564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:19.022530    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:20.913672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:20.913638    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"]
Nov 02 15:41:20.918790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:20.918765    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:20.941175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:20.941148    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"]
Nov 02 15:41:20.950944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:20.945065    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f84ad8-serving-cert\""
Nov 02 15:41:21.063412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.063344    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.063587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.063502    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4k5\" (UniqueName: \"kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.063587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.063549    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.063740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.063724    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.164969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.164887    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4k5\" (UniqueName: \"kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.164969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.164929    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.165166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.165046    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.165213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.165196    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.165766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.165739    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.167686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.167662    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.167799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.167662    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.196631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.196611    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4k5\" (UniqueName: \"kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5\") pod \"maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.234630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.234610    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:21.551237 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:41:21.551208    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c2573c_a19c_42c7_ac42_2ebc0b3cc5a4.slice/crio-c09592c72295a1078e05508a3672e13ad8f4c0bc442d4b05d3d8e93e607fefa3 WatchSource:0}: Error finding container c09592c72295a1078e05508a3672e13ad8f4c0bc442d4b05d3d8e93e607fefa3: Status 404 returned error can't find the container with id c09592c72295a1078e05508a3672e13ad8f4c0bc442d4b05d3d8e93e607fefa3
Nov 02 15:41:21.551340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:21.551250    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"]
Nov 02 15:41:22.292683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:22.292646    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" event={"ID":"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4","Type":"ContainerStarted","Data":"84cc4bc5daf825805fcd81c066a47b8e77f7f144a605cfc9c1fae2d2500d2d50"}
Nov 02 15:41:22.293164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:22.292711    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" event={"ID":"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4","Type":"ContainerStarted","Data":"c09592c72295a1078e05508a3672e13ad8f4c0bc442d4b05d3d8e93e607fefa3"}
Nov 02 15:41:22.293164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:22.292887    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:22.343226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:22.343177    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" podStartSLOduration=2.039143388 podStartE2EDuration="2.343164194s" totalImagesPullingTime="304.020806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:22.341560676 +0000 UTC m=+2734.599630127" watchObservedRunningTime="2026-11-02 15:41:22.343164194 +0000 UTC m=+2734.601233643"
Nov 02 15:41:27.022879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:27.022844    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:27.039131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:27.039100    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:27.349236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:27.349159    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:41:28.306242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:41:28.306208    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:41:36.770066 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:41:36.770022    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:42:06.770551 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:42:06.770512    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:42:09.856776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.856728    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"]
Nov 02 15:42:09.864519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.864494    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:09.910239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.910103    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3805b8-dockercfg-npc9w\""
Nov 02 15:42:09.910239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.910206    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3805b8\""
Nov 02 15:42:09.917919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.917883    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"]
Nov 02 15:42:09.965953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.947684    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:09.965953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.947929    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9b82\" (UniqueName: \"kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82\") pod \"payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:09.977689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.977649    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"]
Nov 02 15:42:09.986152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.986124    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:09.991336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:09.991295    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"]
Nov 02 15:42:10.051915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.049821    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrjt\" (UniqueName: \"kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.051915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.049950    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9b82\" (UniqueName: \"kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82\") pod \"payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:10.051915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.050106    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.051915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.050143    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:10.051915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.050990    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:10.075109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.075075    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9b82\" (UniqueName: \"kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82\") pod \"payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:10.152083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.151997    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.152254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.152195    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrjt\" (UniqueName: \"kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.152781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.152754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.183631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.183607    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:10.183972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.183952    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrjt\" (UniqueName: \"kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.317530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.317498    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:10.498626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.498594    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"]
Nov 02 15:42:10.499226 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:42:10.499192    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fcab3b_ad32_4bbb_9c3d_83afc0f64983.slice/crio-b0d464c380e638dd076862b20ceb81c61b54e044489e086bf06627e892def16c WatchSource:0}: Error finding container b0d464c380e638dd076862b20ceb81c61b54e044489e086bf06627e892def16c: Status 404 returned error can't find the container with id b0d464c380e638dd076862b20ceb81c61b54e044489e086bf06627e892def16c
Nov 02 15:42:10.672856 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:42:10.672829    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ffe29b_a685_4235_9a7f_775b3137887a.slice/crio-7875b0c7f9235445d34669ba61228c4fc6603ebfd167be641879621ddc4666c3 WatchSource:0}: Error finding container 7875b0c7f9235445d34669ba61228c4fc6603ebfd167be641879621ddc4666c3: Status 404 returned error can't find the container with id 7875b0c7f9235445d34669ba61228c4fc6603ebfd167be641879621ddc4666c3
Nov 02 15:42:10.673059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.673037    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"]
Nov 02 15:42:10.693745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.693713    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc" event={"ID":"10fcab3b-ad32-4bbb-9c3d-83afc0f64983","Type":"ContainerStarted","Data":"3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730"}
Nov 02 15:42:10.693892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.693774    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc" event={"ID":"10fcab3b-ad32-4bbb-9c3d-83afc0f64983","Type":"ContainerStarted","Data":"b0d464c380e638dd076862b20ceb81c61b54e044489e086bf06627e892def16c"}
Nov 02 15:42:10.693892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.693827    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:10.695596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.695568    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58" event={"ID":"15ffe29b-a685-4235-9a7f-775b3137887a","Type":"ContainerStarted","Data":"7875b0c7f9235445d34669ba61228c4fc6603ebfd167be641879621ddc4666c3"}
Nov 02 15:42:10.744398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.738809    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"]
Nov 02 15:42:10.746157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.746135    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.766996 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.766972    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-28da58\"/\"openshift-service-ca.crt\""
Nov 02 15:42:10.767253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.767233    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-28da58\"/\"kube-root-ca.crt\""
Nov 02 15:42:10.774202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.774152    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"]
Nov 02 15:42:10.775342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.775296    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc" podStartSLOduration=1.775281415 podStartE2EDuration="1.775281415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:10.767087332 +0000 UTC m=+2783.025156783" watchObservedRunningTime="2026-11-02 15:42:10.775281415 +0000 UTC m=+2783.033350867"
Nov 02 15:42:10.855520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.855496    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-28da58\"/\"rate-test-model-28da58-kserve-self-signed-certs\""
Nov 02 15:42:10.855640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.855522    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-28da58\"/\"default-dockercfg-br85k\""
Nov 02 15:42:10.864590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.864564    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.864890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.864673    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hbzg\" (UniqueName: \"kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.864890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.864698    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.864890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.864730    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.864890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.864795    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.864890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.864856    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966442    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966655    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966752    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hbzg\" (UniqueName: \"kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966785    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966822    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966876    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.966950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.966935    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.967196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.967173    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.967297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.967177    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.969194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.969166    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:10.969410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:10.969390    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:11.005728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.005707    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hbzg\" (UniqueName: \"kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg\") pod \"rate-test-model-28da58-kserve-5d5f695c7c-hcx2h\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") " pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:11.073637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.073601    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:11.429875 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.429847    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"]
Nov 02 15:42:11.430006 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:42:11.429953    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8449d6ca_7eb0_424f_b047_670e48373151.slice/crio-fcda8fe37c025e0b7e3be0f9f9c76d285251d68dbaf8f6584c156081d3b92c28 WatchSource:0}: Error finding container fcda8fe37c025e0b7e3be0f9f9c76d285251d68dbaf8f6584c156081d3b92c28: Status 404 returned error can't find the container with id fcda8fe37c025e0b7e3be0f9f9c76d285251d68dbaf8f6584c156081d3b92c28
Nov 02 15:42:11.705562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.705475    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58" event={"ID":"15ffe29b-a685-4235-9a7f-775b3137887a","Type":"ContainerStarted","Data":"47726d884204321705627080157c716d6ea3cf03213980985850619b810f5d02"}
Nov 02 15:42:11.705728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.705590    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:11.707422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.707391    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" event={"ID":"8449d6ca-7eb0-424f-b047-670e48373151","Type":"ContainerStarted","Data":"eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c"}
Nov 02 15:42:11.707543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.707434    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" event={"ID":"8449d6ca-7eb0-424f-b047-670e48373151","Type":"ContainerStarted","Data":"fcda8fe37c025e0b7e3be0f9f9c76d285251d68dbaf8f6584c156081d3b92c28"}
Nov 02 15:42:11.760067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.760022    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58" podStartSLOduration=2.760008565 podStartE2EDuration="2.760008565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:11.758996423 +0000 UTC m=+2784.017065874" watchObservedRunningTime="2026-11-02 15:42:11.760008565 +0000 UTC m=+2784.018078014"
Nov 02 15:42:11.823408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:11.823335    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" podStartSLOduration=1.823320383 podStartE2EDuration="1.823320383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:11.817146134 +0000 UTC m=+2784.075215583" watchObservedRunningTime="2026-11-02 15:42:11.823320383 +0000 UTC m=+2784.081389834"
Nov 02 15:42:12.074066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:12.073988    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:16.839707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.839671    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"]
Nov 02 15:42:16.844791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.844769    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:16.874463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.874432    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e80c77-dockercfg-w5tv4\""
Nov 02 15:42:16.874463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.874449    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e80c77\""
Nov 02 15:42:16.879932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.879865    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"]
Nov 02 15:42:16.942631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.942595    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:16.942877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:16.942847    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkwf\" (UniqueName: \"kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.048387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.047133    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.048387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.047347    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkwf\" (UniqueName: \"kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.060634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.060600    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.079206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.079170    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkwf\" (UniqueName: \"kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.165700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.165607    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.498846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.498812    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"]
Nov 02 15:42:17.500088 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:42:17.500058    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad2a4d0_c090_4c8e_abd6_4648914827fc.slice/crio-66926e80b2ebf64afa18f3f7bb610d159eab9bb6cebde18b274a71cf5332ff76 WatchSource:0}: Error finding container 66926e80b2ebf64afa18f3f7bb610d159eab9bb6cebde18b274a71cf5332ff76: Status 404 returned error can't find the container with id 66926e80b2ebf64afa18f3f7bb610d159eab9bb6cebde18b274a71cf5332ff76
Nov 02 15:42:17.760738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.760660    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn" event={"ID":"aad2a4d0-c090-4c8e-abd6-4648914827fc","Type":"ContainerStarted","Data":"e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995"}
Nov 02 15:42:17.760738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.760719    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn" event={"ID":"aad2a4d0-c090-4c8e-abd6-4648914827fc","Type":"ContainerStarted","Data":"66926e80b2ebf64afa18f3f7bb610d159eab9bb6cebde18b274a71cf5332ff76"}
Nov 02 15:42:17.760938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.760746    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:17.800981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:17.800937    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn" podStartSLOduration=1.800924195 podStartE2EDuration="1.800924195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:17.799794896 +0000 UTC m=+2790.057864375" watchObservedRunningTime="2026-11-02 15:42:17.800924195 +0000 UTC m=+2790.058993645"
Nov 02 15:42:21.074600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:21.074565    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:21.087583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:21.087555    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:21.710561 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:21.710531    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:42:21.811932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:21.811899    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:42:22.718287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:22.718256    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:42:28.772605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:28.772570    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:42:36.770642 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:42:36.770603    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:42:50.697712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.697667    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"]
Nov 02 15:42:50.702702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.702680    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.734059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.734030    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"]
Nov 02 15:42:50.745405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.740134    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3805b8-serving-cert\""
Nov 02 15:42:50.785864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.785818    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzdc\" (UniqueName: \"kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.786047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.785878    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.786110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.786069    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.786286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.786270    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.887840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.887800    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.888034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.887967    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.888109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.888055    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzdc\" (UniqueName: \"kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.888109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.888087    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.888781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.888754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.890245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.890224    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.890324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.890296    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:50.911465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:50.911441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzdc\" (UniqueName: \"kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc\") pod \"maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:51.017982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:51.017900    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:51.336285 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:42:51.336255    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248d0912_8f3f_4198_a3b7_1596e6da6dec.slice/crio-71253ee7b7a4ea5fee532762f3c94b2ce97a102f96e1b31e1bcde4d7dd653a15 WatchSource:0}: Error finding container 71253ee7b7a4ea5fee532762f3c94b2ce97a102f96e1b31e1bcde4d7dd653a15: Status 404 returned error can't find the container with id 71253ee7b7a4ea5fee532762f3c94b2ce97a102f96e1b31e1bcde4d7dd653a15
Nov 02 15:42:51.336627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:51.336605    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"]
Nov 02 15:42:52.071515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:52.071469    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st" event={"ID":"248d0912-8f3f-4198-a3b7-1596e6da6dec","Type":"ContainerStarted","Data":"cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3"}
Nov 02 15:42:52.071924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:52.071556    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st" event={"ID":"248d0912-8f3f-4198-a3b7-1596e6da6dec","Type":"ContainerStarted","Data":"71253ee7b7a4ea5fee532762f3c94b2ce97a102f96e1b31e1bcde4d7dd653a15"}
Nov 02 15:42:52.071924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:52.071580    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:52.120214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:52.120165    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st" podStartSLOduration=1.813571856 podStartE2EDuration="2.120153267s" totalImagesPullingTime="306.581411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:52.117168447 +0000 UTC m=+2824.375237898" watchObservedRunningTime="2026-11-02 15:42:52.120153267 +0000 UTC m=+2824.378222716"
Nov 02 15:42:58.085577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:58.085546    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:42:58.989452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:58.989355    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"]
Nov 02 15:42:58.994828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:58.994804    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.018689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.018560    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e80c77-serving-cert\""
Nov 02 15:42:59.021690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.021662    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"]
Nov 02 15:42:59.094140 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.094103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.094677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.094168    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.094677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.094376    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.094677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.094468    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcpf\" (UniqueName: \"kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.197373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.196976    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.197373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.197065    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.197373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.197150    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.197373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.197207    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcpf\" (UniqueName: \"kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.201536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.201497    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.202471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.202442    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.204151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.204121    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.225004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.224976    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcpf\" (UniqueName: \"kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf\") pod \"maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.316115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.316030    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:42:59.371893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.371859    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"]
Nov 02 15:42:59.678733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:42:59.678701    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"]
Nov 02 15:42:59.678988 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:42:59.678966    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd334f924_047f_4f60_8ca1_6d8751dd9833.slice/crio-584c3bbb2ea3ff7646e515c28ee5ad33686eb2a590743006a95b53acad911cda WatchSource:0}: Error finding container 584c3bbb2ea3ff7646e515c28ee5ad33686eb2a590743006a95b53acad911cda: Status 404 returned error can't find the container with id 584c3bbb2ea3ff7646e515c28ee5ad33686eb2a590743006a95b53acad911cda
Nov 02 15:43:00.167204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:00.167150    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn" event={"ID":"d334f924-047f-4f60-8ca1-6d8751dd9833","Type":"ContainerStarted","Data":"ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d"}
Nov 02 15:43:00.167722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:00.167221    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn" event={"ID":"d334f924-047f-4f60-8ca1-6d8751dd9833","Type":"ContainerStarted","Data":"584c3bbb2ea3ff7646e515c28ee5ad33686eb2a590743006a95b53acad911cda"}
Nov 02 15:43:00.167722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:00.167252    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:43:00.224091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:00.224038    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn" podStartSLOduration=1.892474033 podStartE2EDuration="2.224024651s" totalImagesPullingTime="331.550618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:00.218817532 +0000 UTC m=+2832.476886977" watchObservedRunningTime="2026-11-02 15:43:00.224024651 +0000 UTC m=+2832.482094101"
Nov 02 15:43:06.181461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.181430    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:43:06.617317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.617239    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"]
Nov 02 15:43:06.622816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.622784    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.646610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.646576    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"]
Nov 02 15:43:06.696829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.696786    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.696989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.696856    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.721481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.721444    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dd139-dockercfg-frn5t\""
Nov 02 15:43:06.723352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.723320    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dd139\""
Nov 02 15:43:06.794287 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:43:06.794241    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:43:06.798617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.798580    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.798918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.798889    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.799461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.799431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.822888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.822864    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:06.944309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:06.944278    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:07.258616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:07.258587    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"]
Nov 02 15:43:07.258931 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:43:07.258780    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8005af5d_1fbd_41ac_9895_91c5e3191f01.slice/crio-ef3b3269320ffa44266ba10833ffc5f679d267869522199a5c78ea59b52c7db1 WatchSource:0}: Error finding container ef3b3269320ffa44266ba10833ffc5f679d267869522199a5c78ea59b52c7db1: Status 404 returned error can't find the container with id ef3b3269320ffa44266ba10833ffc5f679d267869522199a5c78ea59b52c7db1
Nov 02 15:43:08.241515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:08.241479    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn" event={"ID":"8005af5d-1fbd-41ac-9895-91c5e3191f01","Type":"ContainerStarted","Data":"9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b"}
Nov 02 15:43:08.241711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:08.241537    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn" event={"ID":"8005af5d-1fbd-41ac-9895-91c5e3191f01","Type":"ContainerStarted","Data":"ef3b3269320ffa44266ba10833ffc5f679d267869522199a5c78ea59b52c7db1"}
Nov 02 15:43:08.241711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:08.241567    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:08.288337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:08.288288    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn" podStartSLOduration=2.288274438 podStartE2EDuration="2.288274438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:08.284322925 +0000 UTC m=+2840.542392375" watchObservedRunningTime="2026-11-02 15:43:08.288274438 +0000 UTC m=+2840.546343888"
Nov 02 15:43:14.400305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.400271    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg" podUID="1653e611-cfba-42ac-b836-e2564a999d77" containerName="main" containerID="cri-o://44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02" gracePeriod=45
Nov 02 15:43:14.694738 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.694708    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:43:14.801829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.801799    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw\" (UniqueName: \"kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw\") pod \"1653e611-cfba-42ac-b836-e2564a999d77\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") "
Nov 02 15:43:14.801989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.801884    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm\") pod \"1653e611-cfba-42ac-b836-e2564a999d77\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") "
Nov 02 15:43:14.801989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.801932    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs\" (UniqueName: \"kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs\") pod \"1653e611-cfba-42ac-b836-e2564a999d77\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") "
Nov 02 15:43:14.802101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802088    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir\") pod \"1653e611-cfba-42ac-b836-e2564a999d77\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") "
Nov 02 15:43:14.802158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802121    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache\") pod \"1653e611-cfba-42ac-b836-e2564a999d77\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") "
Nov 02 15:43:14.802215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802160    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home\") pod \"1653e611-cfba-42ac-b836-e2564a999d77\" (UID: \"1653e611-cfba-42ac-b836-e2564a999d77\") "
Nov 02 15:43:14.802477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802386    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir" pod "1653e611-cfba-42ac-b836-e2564a999d77" (UID: "1653e611-cfba-42ac-b836-e2564a999d77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.802591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802484    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache" pod "1653e611-cfba-42ac-b836-e2564a999d77" (UID: "1653e611-cfba-42ac-b836-e2564a999d77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.802591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802580    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home" pod "1653e611-cfba-42ac-b836-e2564a999d77" (UID: "1653e611-cfba-42ac-b836-e2564a999d77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.802942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802921    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.803019 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802946    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.803019 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.802960    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.804345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.804319    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw" pod "1653e611-cfba-42ac-b836-e2564a999d77" (UID: "1653e611-cfba-42ac-b836-e2564a999d77"). InnerVolumeSpecName "kube-api-access-lzjzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:43:14.804564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.804545    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm" pod "1653e611-cfba-42ac-b836-e2564a999d77" (UID: "1653e611-cfba-42ac-b836-e2564a999d77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.804564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.804554    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs" pod "1653e611-cfba-42ac-b836-e2564a999d77" (UID: "1653e611-cfba-42ac-b836-e2564a999d77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:43:14.904254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.904219    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjzw\" (UniqueName: \"kubernetes.io/projected/1653e611-cfba-42ac-b836-e2564a999d77-kube-api-access-lzjzw\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.904254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.904251    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1653e611-cfba-42ac-b836-e2564a999d77-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.904254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:14.904262    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1653e611-cfba-42ac-b836-e2564a999d77-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:15.306590 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.306553    5041 generic.go:356] "Generic (PLEG): container finished" podID="1653e611-cfba-42ac-b836-e2564a999d77" containerID="44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02" exitCode=0
Nov 02 15:43:15.306750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.306606    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"
Nov 02 15:43:15.306750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.306633    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg" event={"ID":"1653e611-cfba-42ac-b836-e2564a999d77","Type":"ContainerDied","Data":"44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02"}
Nov 02 15:43:15.306750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.306678    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg" event={"ID":"1653e611-cfba-42ac-b836-e2564a999d77","Type":"ContainerDied","Data":"6fc9c4acdd8c413fccc96969d4d573e4a084b1d465b287e2d7a50a71af2ec6b8"}
Nov 02 15:43:15.306750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.306699    5041 scope.go:122] "RemoveContainer" containerID="44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02"
Nov 02 15:43:15.318807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.318789    5041 scope.go:122] "RemoveContainer" containerID="44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02"
Nov 02 15:43:15.319130 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:43:15.319109    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02\": container with ID starting with 44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02 not found: ID does not exist" containerID="44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02"
Nov 02 15:43:15.319188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.319137    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02"} err="failed to get container status \"44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02\": rpc error: code = NotFound desc = could not find container \"44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02\": container with ID starting with 44d4a29f7da8c4721c017b85a2903edb3922117cc8fd2dffdc86716accb46c02 not found: ID does not exist"
Nov 02 15:43:15.361262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.361241    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"]
Nov 02 15:43:15.371900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.371878    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0a7163/rate-test-model-0a7163-kserve-69b944f6c7-wwdhg"]
Nov 02 15:43:15.890113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:15.890081    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1653e611-cfba-42ac-b836-e2564a999d77" path="/var/lib/kubelet/pods/1653e611-cfba-42ac-b836-e2564a999d77/volumes"
Nov 02 15:43:19.254265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:19.254229    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:43:36.770764 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:43:36.770722    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:43:40.825919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:40.825857    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-143.ec2.internal" podUID="15b2f7b139a98a51bb6e53a9a5605774" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:43:44.247228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:44.247195    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"]
Nov 02 15:43:44.286615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:44.286584    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"]
Nov 02 15:43:47.307442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:47.307403    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"]
Nov 02 15:43:49.247630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.247587    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc" podUID="10fcab3b-ad32-4bbb-9c3d-83afc0f64983" containerName="payload-processing" containerID="cri-o://3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730" gracePeriod=25
Nov 02 15:43:49.287817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.287787    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58" podUID="15ffe29b-a685-4235-9a7f-775b3137887a" containerName="payload-pre-processing" containerID="cri-o://47726d884204321705627080157c716d6ea3cf03213980985850619b810f5d02" gracePeriod=25
Nov 02 15:43:49.572220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.572193    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:43:49.607175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.607143    5041 generic.go:356] "Generic (PLEG): container finished" podID="10fcab3b-ad32-4bbb-9c3d-83afc0f64983" containerID="3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730" exitCode=0
Nov 02 15:43:49.607304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.607185    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"
Nov 02 15:43:49.607304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.607230    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc" event={"ID":"10fcab3b-ad32-4bbb-9c3d-83afc0f64983","Type":"ContainerDied","Data":"3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730"}
Nov 02 15:43:49.607304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.607288    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc" event={"ID":"10fcab3b-ad32-4bbb-9c3d-83afc0f64983","Type":"ContainerDied","Data":"b0d464c380e638dd076862b20ceb81c61b54e044489e086bf06627e892def16c"}
Nov 02 15:43:49.607454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.607320    5041 scope.go:122] "RemoveContainer" containerID="3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730"
Nov 02 15:43:49.608942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.608920    5041 generic.go:356] "Generic (PLEG): container finished" podID="15ffe29b-a685-4235-9a7f-775b3137887a" containerID="47726d884204321705627080157c716d6ea3cf03213980985850619b810f5d02" exitCode=0
Nov 02 15:43:49.609063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.608975    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58" event={"ID":"15ffe29b-a685-4235-9a7f-775b3137887a","Type":"ContainerDied","Data":"47726d884204321705627080157c716d6ea3cf03213980985850619b810f5d02"}
Nov 02 15:43:49.609063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.609005    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58" event={"ID":"15ffe29b-a685-4235-9a7f-775b3137887a","Type":"ContainerDied","Data":"7875b0c7f9235445d34669ba61228c4fc6603ebfd167be641879621ddc4666c3"}
Nov 02 15:43:49.609063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.609026    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7875b0c7f9235445d34669ba61228c4fc6603ebfd167be641879621ddc4666c3"
Nov 02 15:43:49.623221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.623203    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:43:49.623449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.623418    5041 scope.go:122] "RemoveContainer" containerID="3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730"
Nov 02 15:43:49.623745 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:43:49.623716    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730\": container with ID starting with 3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730 not found: ID does not exist" containerID="3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730"
Nov 02 15:43:49.623835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.623749    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730"} err="failed to get container status \"3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730\": rpc error: code = NotFound desc = could not find container \"3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730\": container with ID starting with 3d40f4480c9ec745825a5e629c44eea2840c3154290178dca34266441da1d730 not found: ID does not exist"
Nov 02 15:43:49.631413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.631387    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82\" (UniqueName: \"kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82\") pod \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") "
Nov 02 15:43:49.631534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.631469    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt\" (UniqueName: \"kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt\") pod \"15ffe29b-a685-4235-9a7f-775b3137887a\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") "
Nov 02 15:43:49.631604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.631557    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume\") pod \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\" (UID: \"10fcab3b-ad32-4bbb-9c3d-83afc0f64983\") "
Nov 02 15:43:49.631604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.631598    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume\") pod \"15ffe29b-a685-4235-9a7f-775b3137887a\" (UID: \"15ffe29b-a685-4235-9a7f-775b3137887a\") "
Nov 02 15:43:49.632454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.632058    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume" pod "10fcab3b-ad32-4bbb-9c3d-83afc0f64983" (UID: "10fcab3b-ad32-4bbb-9c3d-83afc0f64983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:43:49.632454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.632189    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume" pod "15ffe29b-a685-4235-9a7f-775b3137887a" (UID: "15ffe29b-a685-4235-9a7f-775b3137887a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:43:49.632454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.632354    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:49.632454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.632421    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ffe29b-a685-4235-9a7f-775b3137887a-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:49.633596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.633576    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82" pod "10fcab3b-ad32-4bbb-9c3d-83afc0f64983" (UID: "10fcab3b-ad32-4bbb-9c3d-83afc0f64983"). InnerVolumeSpecName "kube-api-access-g9b82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:43:49.633842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.633823    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt" pod "15ffe29b-a685-4235-9a7f-775b3137887a" (UID: "15ffe29b-a685-4235-9a7f-775b3137887a"). InnerVolumeSpecName "kube-api-access-4nrjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:43:49.733689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.733653    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrjt\" (UniqueName: \"kubernetes.io/projected/15ffe29b-a685-4235-9a7f-775b3137887a-kube-api-access-4nrjt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:49.733689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.733683    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9b82\" (UniqueName: \"kubernetes.io/projected/10fcab3b-ad32-4bbb-9c3d-83afc0f64983-kube-api-access-g9b82\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:49.950797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.950768    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"]
Nov 02 15:43:49.957052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:49.957024    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-687bcc9975-9zskc"]
Nov 02 15:43:50.180588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.180558    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"]
Nov 02 15:43:50.181390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181359    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1653e611-cfba-42ac-b836-e2564a999d77" containerName="main"
Nov 02 15:43:50.181445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181394    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1653e611-cfba-42ac-b836-e2564a999d77" containerName="main"
Nov 02 15:43:50.181445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181427    5041 cpu_manager.go:414] "containerMap: removing container" podUID="10fcab3b-ad32-4bbb-9c3d-83afc0f64983" containerName="payload-processing"
Nov 02 15:43:50.181445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181435    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fcab3b-ad32-4bbb-9c3d-83afc0f64983" containerName="payload-processing"
Nov 02 15:43:50.181545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181477    5041 cpu_manager.go:414] "containerMap: removing container" podUID="15ffe29b-a685-4235-9a7f-775b3137887a" containerName="payload-pre-processing"
Nov 02 15:43:50.181545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181484    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ffe29b-a685-4235-9a7f-775b3137887a" containerName="payload-pre-processing"
Nov 02 15:43:50.181635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181625    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fcab3b-ad32-4bbb-9c3d-83afc0f64983" containerName="payload-processing"
Nov 02 15:43:50.181670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181642    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ffe29b-a685-4235-9a7f-775b3137887a" containerName="payload-pre-processing"
Nov 02 15:43:50.181670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.181656    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1653e611-cfba-42ac-b836-e2564a999d77" containerName="main"
Nov 02 15:43:50.187035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.186985    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.214484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.213946    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"]
Nov 02 15:43:50.214940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.214914    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5dd139-serving-cert\""
Nov 02 15:43:50.240905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.240877    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.241045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.241021    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.241124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.241050    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmz4\" (UniqueName: \"kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.241124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.241090    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.342142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.342117    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.342601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.342167    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmz4\" (UniqueName: \"kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.342601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.342235    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.342601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.342321    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.342935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.342911    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.344814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.344790    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.344915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.344850    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.370573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.370552    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmz4\" (UniqueName: \"kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4\") pod \"maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.505396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.505322    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:50.621876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.621687    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"
Nov 02 15:43:50.679815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.679781    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"]
Nov 02 15:43:50.685418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.685386    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-76c979b675-lwx58"]
Nov 02 15:43:50.854877 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:43:50.854849    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e0e740_c8aa_4ff0_baac_487596900702.slice/crio-8afd9dd2a103fd21894b57169b31fcefaffde9bca56ec2f68a014699acb54a97 WatchSource:0}: Error finding container 8afd9dd2a103fd21894b57169b31fcefaffde9bca56ec2f68a014699acb54a97: Status 404 returned error can't find the container with id 8afd9dd2a103fd21894b57169b31fcefaffde9bca56ec2f68a014699acb54a97
Nov 02 15:43:50.855470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:50.855447    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"]
Nov 02 15:43:51.631216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:51.631118    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c" event={"ID":"59e0e740-c8aa-4ff0-baac-487596900702","Type":"ContainerStarted","Data":"ca8d1ed77aa4603c9d5022e348a2925a6a36e8a3380184f48604aa3b135fa62e"}
Nov 02 15:43:51.631216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:51.631201    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c" event={"ID":"59e0e740-c8aa-4ff0-baac-487596900702","Type":"ContainerStarted","Data":"8afd9dd2a103fd21894b57169b31fcefaffde9bca56ec2f68a014699acb54a97"}
Nov 02 15:43:51.632049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:51.631336    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:43:51.693610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:51.693553    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c" podStartSLOduration=1.302228368 podStartE2EDuration="1.693539915s" totalImagesPullingTime="391.311547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:51.691912392 +0000 UTC m=+2883.949981840" watchObservedRunningTime="2026-11-02 15:43:51.693539915 +0000 UTC m=+2883.951609404"
Nov 02 15:43:51.891113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:51.891030    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fcab3b-ad32-4bbb-9c3d-83afc0f64983" path="/var/lib/kubelet/pods/10fcab3b-ad32-4bbb-9c3d-83afc0f64983/volumes"
Nov 02 15:43:51.891575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:51.891553    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ffe29b-a685-4235-9a7f-775b3137887a" path="/var/lib/kubelet/pods/15ffe29b-a685-4235-9a7f-775b3137887a/volumes"
Nov 02 15:43:52.307733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.307691    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn" podUID="aad2a4d0-c090-4c8e-abd6-4648914827fc" containerName="payload-pre-processing" containerID="cri-o://e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995" gracePeriod=25
Nov 02 15:43:52.603817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.603790    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:43:52.640810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.640779    5041 generic.go:356] "Generic (PLEG): container finished" podID="aad2a4d0-c090-4c8e-abd6-4648914827fc" containerID="e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995" exitCode=0
Nov 02 15:43:52.641202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.640843    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"
Nov 02 15:43:52.641202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.640870    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn" event={"ID":"aad2a4d0-c090-4c8e-abd6-4648914827fc","Type":"ContainerDied","Data":"e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995"}
Nov 02 15:43:52.641202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.640928    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn" event={"ID":"aad2a4d0-c090-4c8e-abd6-4648914827fc","Type":"ContainerDied","Data":"66926e80b2ebf64afa18f3f7bb610d159eab9bb6cebde18b274a71cf5332ff76"}
Nov 02 15:43:52.641202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.640958    5041 scope.go:122] "RemoveContainer" containerID="e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995"
Nov 02 15:43:52.652023 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.652005    5041 scope.go:122] "RemoveContainer" containerID="e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995"
Nov 02 15:43:52.652268 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:43:52.652251    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995\": container with ID starting with e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995 not found: ID does not exist" containerID="e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995"
Nov 02 15:43:52.652313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.652276    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995"} err="failed to get container status \"e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995\": rpc error: code = NotFound desc = could not find container \"e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995\": container with ID starting with e585a148a723103b3d4b42d26fddd99802d9c34108810388de5aca1a83562995 not found: ID does not exist"
Nov 02 15:43:52.671801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.671778    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume\") pod \"aad2a4d0-c090-4c8e-abd6-4648914827fc\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") "
Nov 02 15:43:52.672012 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.671977    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf\" (UniqueName: \"kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf\") pod \"aad2a4d0-c090-4c8e-abd6-4648914827fc\" (UID: \"aad2a4d0-c090-4c8e-abd6-4648914827fc\") "
Nov 02 15:43:52.672154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.672134    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume" pod "aad2a4d0-c090-4c8e-abd6-4648914827fc" (UID: "aad2a4d0-c090-4c8e-abd6-4648914827fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:43:52.672627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.672584    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad2a4d0-c090-4c8e-abd6-4648914827fc-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:52.674165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.674145    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf" pod "aad2a4d0-c090-4c8e-abd6-4648914827fc" (UID: "aad2a4d0-c090-4c8e-abd6-4648914827fc"). InnerVolumeSpecName "kube-api-access-wfkwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:43:52.773830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.773798    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfkwf\" (UniqueName: \"kubernetes.io/projected/aad2a4d0-c090-4c8e-abd6-4648914827fc-kube-api-access-wfkwf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:43:52.990202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.990167    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"]
Nov 02 15:43:52.996822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:52.996796    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-7f778789ff-j6vgn"]
Nov 02 15:43:53.891319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:53.891276    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad2a4d0-c090-4c8e-abd6-4648914827fc" path="/var/lib/kubelet/pods/aad2a4d0-c090-4c8e-abd6-4648914827fc/volumes"
Nov 02 15:43:57.647163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:43:57.647130    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:44:06.771900 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:44:06.771860    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:44:11.275144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.275098    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"]
Nov 02 15:44:11.276277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.276251    5041 cpu_manager.go:414] "containerMap: removing container" podUID="aad2a4d0-c090-4c8e-abd6-4648914827fc" containerName="payload-pre-processing"
Nov 02 15:44:11.276431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.276276    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad2a4d0-c090-4c8e-abd6-4648914827fc" containerName="payload-pre-processing"
Nov 02 15:44:11.276563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.276547    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad2a4d0-c090-4c8e-abd6-4648914827fc" containerName="payload-pre-processing"
Nov 02 15:44:11.282158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.282134    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.315978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.315905    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"]
Nov 02 15:44:11.376413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.376386    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e80c77-openshift-default-dockercfg-b7nkf\""
Nov 02 15:44:11.400464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400430    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400490    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400669    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400700    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400814    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400839    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400878    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchsd\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.400960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.400951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.401016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.401001    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.501867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.501830    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.501867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.501866    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.501939    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.501967    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502108    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchsd\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502190    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502267    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502339    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502347    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502456    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502505    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502676    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502708    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.502781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.502751    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.504550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.504529    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.504893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.504876    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.537103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.537044    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.537103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.537049    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchsd\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.599576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.599554    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:11.930812 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:44:11.930781    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a50fcd5_8c60_4913_afad_6174f9d38f2b.slice/crio-b954c25f1cd04674ab4947783a36ad917ba6cb9c0a01316f3b6e31a1f554df26 WatchSource:0}: Error finding container b954c25f1cd04674ab4947783a36ad917ba6cb9c0a01316f3b6e31a1f554df26: Status 404 returned error can't find the container with id b954c25f1cd04674ab4947783a36ad917ba6cb9c0a01316f3b6e31a1f554df26
Nov 02 15:44:11.931924 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.931895    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"]
Nov 02 15:44:11.933297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.933248    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:11.933401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.933329    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:11.933401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:11.933393    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:12.832497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:12.832456    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" event={"ID":"9a50fcd5-8c60-4913-afad-6174f9d38f2b","Type":"ContainerStarted","Data":"a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc"}
Nov 02 15:44:12.833007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:12.832524    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" event={"ID":"9a50fcd5-8c60-4913-afad-6174f9d38f2b","Type":"ContainerStarted","Data":"b954c25f1cd04674ab4947783a36ad917ba6cb9c0a01316f3b6e31a1f554df26"}
Nov 02 15:44:12.907024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:12.906955    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podStartSLOduration=1.906930888 podStartE2EDuration="1.906930888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:12.891087842 +0000 UTC m=+2905.149157292" watchObservedRunningTime="2026-11-02 15:44:12.906930888 +0000 UTC m=+2905.165000341"
Nov 02 15:44:13.600560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:13.600521    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:13.602840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:13.602768    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:44:13.603001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:13.602845    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Nov 02 15:44:14.600122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:14.600086    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:44:14.600560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:14.600147    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Nov 02 15:44:15.600347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:15.600308    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:44:15.600810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:15.600406    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Nov 02 15:44:16.600181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:16.600146    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:44:16.600403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:16.600196    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Nov 02 15:44:17.600762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:17.600720    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:44:17.601200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:17.600799    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Nov 02 15:44:18.600072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:18.600028    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:44:18.600285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:18.600088    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Nov 02 15:44:19.019935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.019893    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"]
Nov 02 15:44:19.024020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.023995    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.037584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.037562    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3805b8-openshift-default-dockercfg-h5w4z\""
Nov 02 15:44:19.054464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.054434    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"]
Nov 02 15:44:19.109312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109272    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109413    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109445    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109500    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109546    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109702    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109766    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8jg\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109815    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.109967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.109860    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211197    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211256    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8jg\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211278    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211303    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211448    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211476    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211498    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211550    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211764    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.211949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.211789    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.212098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.212072    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.212296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.212263    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.212452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.212297    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.213808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.213786    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.214073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.214054    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.234749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.234724    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.235727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.235706    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8jg\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.340304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.340247    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:19.605539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.605455    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:19.606878 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.606856    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:19.607461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.607439    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:44:19.677001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.676972    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"]
Nov 02 15:44:19.677390 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:44:19.677344    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e080a3f_5ef7_4c1f_8a1b_d14f6f1f1425.slice/crio-aec3352f0c5faa26befe4b51e8fb0ff304213e95e429c59966978d31f8322205 WatchSource:0}: Error finding container aec3352f0c5faa26befe4b51e8fb0ff304213e95e429c59966978d31f8322205: Status 404 returned error can't find the container with id aec3352f0c5faa26befe4b51e8fb0ff304213e95e429c59966978d31f8322205
Nov 02 15:44:19.680219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.680182    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:19.680313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.680248    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:19.680313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.680278    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:19.898039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.897940    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" event={"ID":"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425","Type":"ContainerStarted","Data":"ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938"}
Nov 02 15:44:19.898039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.898003    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" event={"ID":"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425","Type":"ContainerStarted","Data":"aec3352f0c5faa26befe4b51e8fb0ff304213e95e429c59966978d31f8322205"}
Nov 02 15:44:19.956231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:19.956183    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podStartSLOduration=1.956169627 podStartE2EDuration="1.956169627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:19.948901897 +0000 UTC m=+2912.206971351" watchObservedRunningTime="2026-11-02 15:44:19.956169627 +0000 UTC m=+2912.214239076"
Nov 02 15:44:20.341006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:20.340971    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:20.342121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:20.342095    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 15:44:20.342235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:20.342148    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 15:44:21.341397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:21.341336    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 15:44:21.341829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:21.341436    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 15:44:22.341586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:22.341542    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 15:44:22.341998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:22.341601    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 15:44:23.341190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:23.341143    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 15:44:23.341432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:23.341222    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 15:44:24.341204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:24.341166    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 15:44:24.341656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:24.341240    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 15:44:25.341191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:25.341156    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 15:44:25.341403 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:25.341219    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 15:44:26.387688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:26.387628    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:26.388830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:26.388802    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:26.389749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:26.389728    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:44:30.706938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:30.706861    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"]
Nov 02 15:44:36.243868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.243827    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"]
Nov 02 15:44:36.248792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.248768    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.277117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.277077    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e80c77-dockercfg-pjkdp\""
Nov 02 15:44:36.280596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.280566    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e80c77\""
Nov 02 15:44:36.283605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.283571    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"]
Nov 02 15:44:36.337851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.337825    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f987k\" (UniqueName: \"kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k\") pod \"payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.337993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.337970    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.439207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.439179    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.439373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.439344    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f987k\" (UniqueName: \"kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k\") pod \"payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.439897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.439876    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.462755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.462725    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f987k\" (UniqueName: \"kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k\") pod \"payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.564506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.564441    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:36.770144 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:44:36.770108    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:44:36.873054 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:44:36.873018    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a80097a_9d77_41e7_b551_3a71731000ff.slice/crio-c7229ef9d2d3343edde2ef3d57244d92a8351ced79e816ce922a49c506e1da2a WatchSource:0}: Error finding container c7229ef9d2d3343edde2ef3d57244d92a8351ced79e816ce922a49c506e1da2a: Status 404 returned error can't find the container with id c7229ef9d2d3343edde2ef3d57244d92a8351ced79e816ce922a49c506e1da2a
Nov 02 15:44:36.873487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.873465    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"]
Nov 02 15:44:36.875071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:36.875050    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:44:37.079487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:37.079458    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" event={"ID":"1a80097a-9d77-41e7-b551-3a71731000ff","Type":"ContainerStarted","Data":"c7229ef9d2d3343edde2ef3d57244d92a8351ced79e816ce922a49c506e1da2a"}
Nov 02 15:44:39.103408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:39.103344    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" event={"ID":"1a80097a-9d77-41e7-b551-3a71731000ff","Type":"ContainerStarted","Data":"2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"}
Nov 02 15:44:39.103870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:39.103532    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:39.152487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:39.152437    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" podStartSLOduration=1.572627631 podStartE2EDuration="3.152422725s" totalImagesPullingTime="1.579795094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:39.148887562 +0000 UTC m=+2931.406957011" watchObservedRunningTime="2026-11-02 15:44:39.152422725 +0000 UTC m=+2931.410492175"
Nov 02 15:44:45.733503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:45.733464    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw" podUID="d09f58ea-e7e2-4c1f-b506-27bbffd37f73" containerName="main" containerID="cri-o://3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985" gracePeriod=45
Nov 02 15:44:46.015573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.015548    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:44:46.162816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.162780    5041 generic.go:356] "Generic (PLEG): container finished" podID="d09f58ea-e7e2-4c1f-b506-27bbffd37f73" containerID="3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985" exitCode=0
Nov 02 15:44:46.162967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.162821    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"
Nov 02 15:44:46.162967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.162865    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw" event={"ID":"d09f58ea-e7e2-4c1f-b506-27bbffd37f73","Type":"ContainerDied","Data":"3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985"}
Nov 02 15:44:46.162967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.162909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw" event={"ID":"d09f58ea-e7e2-4c1f-b506-27bbffd37f73","Type":"ContainerDied","Data":"40a0d8b5f5a017c637407aa949889eba76211e6c6f81941a1be9780a2921683a"}
Nov 02 15:44:46.162967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.162925    5041 scope.go:122] "RemoveContainer" containerID="3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985"
Nov 02 15:44:46.168433 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168412    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs\") pod \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") "
Nov 02 15:44:46.168528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168474    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir\") pod \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") "
Nov 02 15:44:46.168596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168574    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt\" (UniqueName: \"kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt\") pod \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") "
Nov 02 15:44:46.168683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168665    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache\") pod \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") "
Nov 02 15:44:46.168752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168714    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home\") pod \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") "
Nov 02 15:44:46.168752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168725    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir" pod "d09f58ea-e7e2-4c1f-b506-27bbffd37f73" (UID: "d09f58ea-e7e2-4c1f-b506-27bbffd37f73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:44:46.168869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168781    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm\") pod \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\" (UID: \"d09f58ea-e7e2-4c1f-b506-27bbffd37f73\") "
Nov 02 15:44:46.169008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.168983    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache" pod "d09f58ea-e7e2-4c1f-b506-27bbffd37f73" (UID: "d09f58ea-e7e2-4c1f-b506-27bbffd37f73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:44:46.169100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.169023    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home" pod "d09f58ea-e7e2-4c1f-b506-27bbffd37f73" (UID: "d09f58ea-e7e2-4c1f-b506-27bbffd37f73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:44:46.169495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.169471    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.169614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.169500    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.169614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.169515    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.170744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.170717    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs" pod "d09f58ea-e7e2-4c1f-b506-27bbffd37f73" (UID: "d09f58ea-e7e2-4c1f-b506-27bbffd37f73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:44:46.170861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.170831    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm" pod "d09f58ea-e7e2-4c1f-b506-27bbffd37f73" (UID: "d09f58ea-e7e2-4c1f-b506-27bbffd37f73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:44:46.170963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.170946    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt" pod "d09f58ea-e7e2-4c1f-b506-27bbffd37f73" (UID: "d09f58ea-e7e2-4c1f-b506-27bbffd37f73"). InnerVolumeSpecName "kube-api-access-8rvrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:46.174296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.174272    5041 scope.go:122] "RemoveContainer" containerID="3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985"
Nov 02 15:44:46.174617 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:44:46.174587    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985\": container with ID starting with 3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985 not found: ID does not exist" containerID="3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985"
Nov 02 15:44:46.174688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.174614    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985"} err="failed to get container status \"3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985\": rpc error: code = NotFound desc = could not find container \"3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985\": container with ID starting with 3dedc43180ae28891bf929198912e303e7d7b4433abf3102597c48595410f985 not found: ID does not exist"
Nov 02 15:44:46.271482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.271403    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.271482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.271434    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvrt\" (UniqueName: \"kubernetes.io/projected/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-kube-api-access-8rvrt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.271482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.271449    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d09f58ea-e7e2-4c1f-b506-27bbffd37f73-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.507395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.507349    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"]
Nov 02 15:44:46.519647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.519618    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8a3fb0/auth-test-model-8a3fb0-kserve-76c8896f77-rk9nw"]
Nov 02 15:44:46.646058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:46.645984    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"]
Nov 02 15:44:47.891581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:47.891546    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09f58ea-e7e2-4c1f-b506-27bbffd37f73" path="/var/lib/kubelet/pods/d09f58ea-e7e2-4c1f-b506-27bbffd37f73/volumes"
Nov 02 15:44:49.191049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:49.191015    5041 generic.go:356] "Generic (PLEG): container finished" podID="1a80097a-9d77-41e7-b551-3a71731000ff" containerID="2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f" exitCode=1
Nov 02 15:44:49.191475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:49.191100    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" event={"ID":"1a80097a-9d77-41e7-b551-3a71731000ff","Type":"ContainerDied","Data":"2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"}
Nov 02 15:44:49.191601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:49.191583    5041 scope.go:122] "RemoveContainer" containerID="2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"
Nov 02 15:44:50.200824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:50.200789    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" event={"ID":"1a80097a-9d77-41e7-b551-3a71731000ff","Type":"ContainerStarted","Data":"df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea"}
Nov 02 15:44:50.201208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:50.200989    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:44:51.646903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:51.646847    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn" podUID="8005af5d-1fbd-41ac-9895-91c5e3191f01" containerName="payload-pre-processing" containerID="cri-o://9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b" gracePeriod=25
Nov 02 15:44:51.960770 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:51.960743    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:44:52.039354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.039323    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d\") pod \"8005af5d-1fbd-41ac-9895-91c5e3191f01\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") "
Nov 02 15:44:52.039534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.039386    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume\") pod \"8005af5d-1fbd-41ac-9895-91c5e3191f01\" (UID: \"8005af5d-1fbd-41ac-9895-91c5e3191f01\") "
Nov 02 15:44:52.039851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.039822    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume" pod "8005af5d-1fbd-41ac-9895-91c5e3191f01" (UID: "8005af5d-1fbd-41ac-9895-91c5e3191f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:44:52.040032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.040017    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8005af5d-1fbd-41ac-9895-91c5e3191f01-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:52.041612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.041588    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d" pod "8005af5d-1fbd-41ac-9895-91c5e3191f01" (UID: "8005af5d-1fbd-41ac-9895-91c5e3191f01"). InnerVolumeSpecName "kube-api-access-wmf2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:52.141726 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.141694    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/8005af5d-1fbd-41ac-9895-91c5e3191f01-kube-api-access-wmf2d\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:44:52.219969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.219937    5041 generic.go:356] "Generic (PLEG): container finished" podID="8005af5d-1fbd-41ac-9895-91c5e3191f01" containerID="9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b" exitCode=0
Nov 02 15:44:52.220069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.219988    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"
Nov 02 15:44:52.220069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.220024    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn" event={"ID":"8005af5d-1fbd-41ac-9895-91c5e3191f01","Type":"ContainerDied","Data":"9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b"}
Nov 02 15:44:52.220147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.220079    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn" event={"ID":"8005af5d-1fbd-41ac-9895-91c5e3191f01","Type":"ContainerDied","Data":"ef3b3269320ffa44266ba10833ffc5f679d267869522199a5c78ea59b52c7db1"}
Nov 02 15:44:52.220147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.220103    5041 scope.go:122] "RemoveContainer" containerID="9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b"
Nov 02 15:44:52.231481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.231465    5041 scope.go:122] "RemoveContainer" containerID="9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b"
Nov 02 15:44:52.231721 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:44:52.231701    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b\": container with ID starting with 9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b not found: ID does not exist" containerID="9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b"
Nov 02 15:44:52.231774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.231728    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b"} err="failed to get container status \"9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b\": rpc error: code = NotFound desc = could not find container \"9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b\": container with ID starting with 9542e8392ecd07e21a9f86e3161bb07a0039372a789904217522f83ee04efa5b not found: ID does not exist"
Nov 02 15:44:52.274345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.274321    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"]
Nov 02 15:44:52.283937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:52.283916    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7fd49fb859-jqjxn"]
Nov 02 15:44:53.890422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:53.890358    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8005af5d-1fbd-41ac-9895-91c5e3191f01" path="/var/lib/kubelet/pods/8005af5d-1fbd-41ac-9895-91c5e3191f01/volumes"
Nov 02 15:44:56.336505 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.336450    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"]
Nov 02 15:44:56.337874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.337847    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8005af5d-1fbd-41ac-9895-91c5e3191f01" containerName="payload-pre-processing"
Nov 02 15:44:56.337961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.337876    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8005af5d-1fbd-41ac-9895-91c5e3191f01" containerName="payload-pre-processing"
Nov 02 15:44:56.337961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.337936    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d09f58ea-e7e2-4c1f-b506-27bbffd37f73" containerName="main"
Nov 02 15:44:56.337961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.337944    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09f58ea-e7e2-4c1f-b506-27bbffd37f73" containerName="main"
Nov 02 15:44:56.338179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.338163    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8005af5d-1fbd-41ac-9895-91c5e3191f01" containerName="payload-pre-processing"
Nov 02 15:44:56.338216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.338195    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09f58ea-e7e2-4c1f-b506-27bbffd37f73" containerName="main"
Nov 02 15:44:56.344149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.344127    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.388257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.388225    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"]
Nov 02 15:44:56.389396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389350    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.389692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389649    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.389865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389747    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.389865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389823    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.389865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389849    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.390195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389872    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.390195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.389938    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskr5\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.390195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.390002    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.390195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.390027    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.490995    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491163    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491193    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491214    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491261    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskr5\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491303    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491319    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491430    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491458    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491568    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491564    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491847    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.491975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.491869    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.492111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.492095    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.493544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.493525    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.493766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.493746    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.513108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.513080    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskr5\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.513108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.513099    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.659495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.659436    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:56.982326 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:44:56.982296    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b76898c_ef4e_45a3_bec8_88e611382ee7.slice/crio-ef5fa3f7b80e69e41f09b60226c7183fc9a4adaaa5f7c64cb287649c1760d92d WatchSource:0}: Error finding container ef5fa3f7b80e69e41f09b60226c7183fc9a4adaaa5f7c64cb287649c1760d92d: Status 404 returned error can't find the container with id ef5fa3f7b80e69e41f09b60226c7183fc9a4adaaa5f7c64cb287649c1760d92d
Nov 02 15:44:56.982949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.982923    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"]
Nov 02 15:44:56.985396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.985344    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:56.985503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.985441    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:56.985503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:56.985489    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:57.270100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:57.270015    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" event={"ID":"3b76898c-ef4e-45a3-bec8-88e611382ee7","Type":"ContainerStarted","Data":"a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7"}
Nov 02 15:44:57.270100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:57.270071    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" event={"ID":"3b76898c-ef4e-45a3-bec8-88e611382ee7","Type":"ContainerStarted","Data":"ef5fa3f7b80e69e41f09b60226c7183fc9a4adaaa5f7c64cb287649c1760d92d"}
Nov 02 15:44:57.314382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:57.314316    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podStartSLOduration=1.314301395 podStartE2EDuration="1.314301395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:57.31140591 +0000 UTC m=+2949.569475362" watchObservedRunningTime="2026-11-02 15:44:57.314301395 +0000 UTC m=+2949.572370845"
Nov 02 15:44:57.663592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:57.663494    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:44:58.662642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:58.662607    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Nov 02 15:44:58.662847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:58.662660    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Nov 02 15:44:59.660517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:59.660478    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Nov 02 15:44:59.660940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:44:59.660555    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Nov 02 15:45:00.374575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.374541    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"]
Nov 02 15:45:00.400146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.400113    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"]
Nov 02 15:45:00.403164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.401453    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.435765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.418185    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:00.450479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.450444    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.454673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.439792    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"]
Nov 02 15:45:00.454673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.453576    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"]
Nov 02 15:45:00.463394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.463351    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"]
Nov 02 15:45:00.463515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.463486    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.472553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.472530    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"]
Nov 02 15:45:00.472740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.472725    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"]
Nov 02 15:45:00.472883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.472636    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.476260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.476242    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"]
Nov 02 15:45:00.478179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.478155    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv92r\" (UniqueName: \"kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.478489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.478465    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.580328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.580283    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nmm\" (UniqueName: \"kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.580536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.580389    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktkd\" (UniqueName: \"kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.580536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.580433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.580674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.580629    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.581184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.581153    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmnt2\" (UniqueName: \"kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2\") pod \"maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.581303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.581200    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.581389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.581340    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.581462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.581425    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv92r\" (UniqueName: \"kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.581566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.581543    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.648430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.647883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv92r\" (UniqueName: \"kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.659853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.659817    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Nov 02 15:45:00.660038 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.659878    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Nov 02 15:45:00.683804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.683381    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmnt2\" (UniqueName: \"kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2\") pod \"maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.683804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.683434    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.683804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.683541    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.684458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.684422    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.684458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.684427    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.684690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.684661    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nmm\" (UniqueName: \"kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.685015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.684990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktkd\" (UniqueName: \"kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.685128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.685112    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.685947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.685919    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.710222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.710187    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nmm\" (UniqueName: \"kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.710539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.710507    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktkd\" (UniqueName: \"kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:00.710661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.710606    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmnt2\" (UniqueName: \"kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2\") pod \"maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.758973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.758771    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:00.771267 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.771241    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:00.782887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.782860    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:00.792183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:00.792158    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:01.168757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.167901    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"]
Nov 02 15:45:01.224395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.212469    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:45:01.392730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.392696    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"]
Nov 02 15:45:01.453649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.453552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerStarted","Data":"56317efc37ddf201b76d36fe2794393a259224e17e7b88fd2ce15c4d8300b6cb"}
Nov 02 15:45:01.567946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.567919    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"]
Nov 02 15:45:01.609273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.609220    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"]
Nov 02 15:45:01.663849 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:01.644463    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc7b279_712c_4006_b755_23ed706cfd9c.slice/crio-fc8ba48b46fd2c22aa5955d7cadfc19bcc0316429171828ccbb819c37d916d11 WatchSource:0}: Error finding container fc8ba48b46fd2c22aa5955d7cadfc19bcc0316429171828ccbb819c37d916d11: Status 404 returned error can't find the container with id fc8ba48b46fd2c22aa5955d7cadfc19bcc0316429171828ccbb819c37d916d11
Nov 02 15:45:01.683418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.664003    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"]
Nov 02 15:45:01.687237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.687203    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Nov 02 15:45:01.699682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:01.699644    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Nov 02 15:45:02.466193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.466164    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerStarted","Data":"11f3405a6d93cc27fa9a3f9fd19279e816a4695be31e7d1dda1810f46789cf96"}
Nov 02 15:45:02.466320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.466223    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerStarted","Data":"fc8ba48b46fd2c22aa5955d7cadfc19bcc0316429171828ccbb819c37d916d11"}
Nov 02 15:45:02.468146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.468119    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerStarted","Data":"813836a7a0f44696788d6b3ccf70e00bdc0ebdb0d16e80afb6a249956a557a32"}
Nov 02 15:45:02.468269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.468170    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerStarted","Data":"9033e3ccf19e8c1fb6a46ded8e648bec7f10194b3477908d980973005ade8006"}
Nov 02 15:45:02.469786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.469757    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerStarted","Data":"ea5899c9093c5d85cf6d491de041a981928b14a7d0e7a918aed5691a505df3de"}
Nov 02 15:45:02.471265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.471238    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerStarted","Data":"d32aa85e6f62535033e52cb4d4816dee5c1eec3db3dd39b402eb9f91d92d6919"}
Nov 02 15:45:02.504527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.504484    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" podStartSLOduration=2.50447327 podStartE2EDuration="2.50447327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.503609402 +0000 UTC m=+2954.761678849" watchObservedRunningTime="2026-11-02 15:45:02.50447327 +0000 UTC m=+2954.762542720"
Nov 02 15:45:02.561498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.561445    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" podStartSLOduration=2.56142942 podStartE2EDuration="2.56142942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.55687022 +0000 UTC m=+2954.814939671" watchObservedRunningTime="2026-11-02 15:45:02.56142942 +0000 UTC m=+2954.819498871"
Nov 02 15:45:02.661164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.661014    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Nov 02 15:45:02.661164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.661082    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Nov 02 15:45:02.692760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.692711    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" podStartSLOduration=2.692697677 podStartE2EDuration="2.692697677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.632560121 +0000 UTC m=+2954.890629576" watchObservedRunningTime="2026-11-02 15:45:02.692697677 +0000 UTC m=+2954.950767127"
Nov 02 15:45:02.694055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:02.694022    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" podStartSLOduration=2.6940136089999998 podStartE2EDuration="2.694013609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.687092198 +0000 UTC m=+2954.945161644" watchObservedRunningTime="2026-11-02 15:45:02.694013609 +0000 UTC m=+2954.952083058"
Nov 02 15:45:03.480752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.480720    5041 generic.go:356] "Generic (PLEG): container finished" podID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerID="11f3405a6d93cc27fa9a3f9fd19279e816a4695be31e7d1dda1810f46789cf96" exitCode=60
Nov 02 15:45:03.480944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.480812    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerDied","Data":"11f3405a6d93cc27fa9a3f9fd19279e816a4695be31e7d1dda1810f46789cf96"}
Nov 02 15:45:03.481113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.481096    5041 scope.go:122] "RemoveContainer" containerID="11f3405a6d93cc27fa9a3f9fd19279e816a4695be31e7d1dda1810f46789cf96"
Nov 02 15:45:03.482676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.482651    5041 generic.go:356] "Generic (PLEG): container finished" podID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerID="813836a7a0f44696788d6b3ccf70e00bdc0ebdb0d16e80afb6a249956a557a32" exitCode=60
Nov 02 15:45:03.482776 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.482728    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerDied","Data":"813836a7a0f44696788d6b3ccf70e00bdc0ebdb0d16e80afb6a249956a557a32"}
Nov 02 15:45:03.482978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.482951    5041 scope.go:122] "RemoveContainer" containerID="813836a7a0f44696788d6b3ccf70e00bdc0ebdb0d16e80afb6a249956a557a32"
Nov 02 15:45:03.484902 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.484877    5041 generic.go:356] "Generic (PLEG): container finished" podID="5bc27548-b535-4913-b1d3-9debc341f230" containerID="086a58e3a7086f43e6e492435d9ef8d9eb742b87cc6eeeb5b213ae18206b6e38" exitCode=60
Nov 02 15:45:03.485006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.484990    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerDied","Data":"086a58e3a7086f43e6e492435d9ef8d9eb742b87cc6eeeb5b213ae18206b6e38"}
Nov 02 15:45:03.485254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.485206    5041 scope.go:122] "RemoveContainer" containerID="086a58e3a7086f43e6e492435d9ef8d9eb742b87cc6eeeb5b213ae18206b6e38"
Nov 02 15:45:03.487156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.487077    5041 generic.go:356] "Generic (PLEG): container finished" podID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerID="d32aa85e6f62535033e52cb4d4816dee5c1eec3db3dd39b402eb9f91d92d6919" exitCode=60
Nov 02 15:45:03.487212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.487158    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerDied","Data":"d32aa85e6f62535033e52cb4d4816dee5c1eec3db3dd39b402eb9f91d92d6919"}
Nov 02 15:45:03.487382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.487351    5041 scope.go:122] "RemoveContainer" containerID="d32aa85e6f62535033e52cb4d4816dee5c1eec3db3dd39b402eb9f91d92d6919"
Nov 02 15:45:03.667082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.667052    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:45:03.667610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.667588    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:45:03.668467 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:03.668441    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:45:04.194649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.194612    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"]
Nov 02 15:45:04.210386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.210207    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.230006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.229973    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"]
Nov 02 15:45:04.325064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.324981    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325025    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325175    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr72q\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325277    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325344    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325449    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325479    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325636    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.325727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.325703    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.427580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.427543    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.427790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.427697    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.427790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.427744    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.427790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.427783    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.427991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.427907    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.427991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.427964    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428001    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428015    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428073    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428125    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428200    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428226    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr72q\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.428470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.428227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.431836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.429241    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.447415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.447351    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.448608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.448583    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.449724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.449699    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.450557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.450539    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr72q\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.498463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.498423    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerStarted","Data":"aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172"}
Nov 02 15:45:04.503420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.500569    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerStarted","Data":"073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0"}
Nov 02 15:45:04.503420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.502901    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerStarted","Data":"f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21"}
Nov 02 15:45:04.505149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.505123    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerStarted","Data":"cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630"}
Nov 02 15:45:04.541567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.541515    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:04.878792 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:04.878757    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880ceacb_6c92_4c7a_b0ef_7c2d72b2697a.slice/crio-e5b416abb6174d975707fd43e741a911033b9f97457c280727c0b1c4639a23a6 WatchSource:0}: Error finding container e5b416abb6174d975707fd43e741a911033b9f97457c280727c0b1c4639a23a6: Status 404 returned error can't find the container with id e5b416abb6174d975707fd43e741a911033b9f97457c280727c0b1c4639a23a6
Nov 02 15:45:04.880932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.879907    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"]
Nov 02 15:45:04.881908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.881872    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:04.882137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.881957    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:04.882137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:04.881999    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.540764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.540728    5041 generic.go:356] "Generic (PLEG): container finished" podID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerID="aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172" exitCode=60
Nov 02 15:45:05.541275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.540821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerDied","Data":"aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172"}
Nov 02 15:45:05.541275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.540900    5041 scope.go:122] "RemoveContainer" containerID="813836a7a0f44696788d6b3ccf70e00bdc0ebdb0d16e80afb6a249956a557a32"
Nov 02 15:45:05.541275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.541137    5041 scope.go:122] "RemoveContainer" containerID="aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172"
Nov 02 15:45:05.541620 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:05.541591    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc_odh-ai-gateway-infra(0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56"
Nov 02 15:45:05.543342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.543322    5041 generic.go:356] "Generic (PLEG): container finished" podID="5bc27548-b535-4913-b1d3-9debc341f230" containerID="073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0" exitCode=60
Nov 02 15:45:05.543462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.543391    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerDied","Data":"073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0"}
Nov 02 15:45:05.543729 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.543710    5041 scope.go:122] "RemoveContainer" containerID="073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0"
Nov 02 15:45:05.544093 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:05.544073    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq_odh-ai-gateway-infra(5bc27548-b535-4913-b1d3-9debc341f230)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" podUID="5bc27548-b535-4913-b1d3-9debc341f230"
Nov 02 15:45:05.545465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.545442    5041 generic.go:356] "Generic (PLEG): container finished" podID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerID="f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21" exitCode=60
Nov 02 15:45:05.545579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.545558    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerDied","Data":"f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21"}
Nov 02 15:45:05.545756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.545731    5041 scope.go:122] "RemoveContainer" containerID="f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21"
Nov 02 15:45:05.546022 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:05.546002    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2_odh-ai-gateway-infra(a75779b2-f411-4bdc-8b75-4be7f7166687)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687"
Nov 02 15:45:05.547983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.547961    5041 generic.go:356] "Generic (PLEG): container finished" podID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerID="cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630" exitCode=60
Nov 02 15:45:05.548071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.548037    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerDied","Data":"cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630"}
Nov 02 15:45:05.548306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.548288    5041 scope.go:122] "RemoveContainer" containerID="cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630"
Nov 02 15:45:05.548569 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:05.548544    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn_odh-ai-gateway-infra(5fc7b279-712c-4006-b755-23ed706cfd9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c"
Nov 02 15:45:05.549855 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.549833    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" event={"ID":"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a","Type":"ContainerStarted","Data":"b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928"}
Nov 02 15:45:05.549957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.549875    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" event={"ID":"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a","Type":"ContainerStarted","Data":"e5b416abb6174d975707fd43e741a911033b9f97457c280727c0b1c4639a23a6"}
Nov 02 15:45:05.557207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.557045    5041 scope.go:122] "RemoveContainer" containerID="086a58e3a7086f43e6e492435d9ef8d9eb742b87cc6eeeb5b213ae18206b6e38"
Nov 02 15:45:05.573558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.573538    5041 scope.go:122] "RemoveContainer" containerID="d32aa85e6f62535033e52cb4d4816dee5c1eec3db3dd39b402eb9f91d92d6919"
Nov 02 15:45:05.590622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.590604    5041 scope.go:122] "RemoveContainer" containerID="11f3405a6d93cc27fa9a3f9fd19279e816a4695be31e7d1dda1810f46789cf96"
Nov 02 15:45:05.771303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.771248    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podStartSLOduration=1.7712340100000001 podStartE2EDuration="1.77123401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:05.766307406 +0000 UTC m=+2958.024376856" watchObservedRunningTime="2026-11-02 15:45:05.77123401 +0000 UTC m=+2958.029303459"
Nov 02 15:45:05.990848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.990814    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6906-50b046"]
Nov 02 15:45:05.997544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:05.997520    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:06.010322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.010281    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcqh6\""
Nov 02 15:45:06.040392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.014050    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6906-50b046"]
Nov 02 15:45:06.157188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.157150    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9bx\" (UniqueName: \"kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx\") pod \"test-curl-6906-50b046\" (UID: \"b386a213-c422-45ec-a42c-588610440dd7\") " pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:06.258956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.258861    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9bx\" (UniqueName: \"kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx\") pod \"test-curl-6906-50b046\" (UID: \"b386a213-c422-45ec-a42c-588610440dd7\") " pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:06.280159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.280127    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9bx\" (UniqueName: \"kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx\") pod \"test-curl-6906-50b046\" (UID: \"b386a213-c422-45ec-a42c-588610440dd7\") " pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:06.311458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.311434    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:06.542163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.542086    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:06.543539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.543512    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 15:45:06.543644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.543567    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 15:45:06.562252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.562195    5041 scope.go:122] "RemoveContainer" containerID="cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630"
Nov 02 15:45:06.562664 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:06.562633    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn_odh-ai-gateway-infra(5fc7b279-712c-4006-b755-23ed706cfd9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c"
Nov 02 15:45:06.564461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.564437    5041 scope.go:122] "RemoveContainer" containerID="aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172"
Nov 02 15:45:06.564729 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:06.564704    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc_odh-ai-gateway-infra(0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56"
Nov 02 15:45:06.566261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.566233    5041 scope.go:122] "RemoveContainer" containerID="073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0"
Nov 02 15:45:06.566516 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:06.566480    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq_odh-ai-gateway-infra(5bc27548-b535-4913-b1d3-9debc341f230)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" podUID="5bc27548-b535-4913-b1d3-9debc341f230"
Nov 02 15:45:06.568004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.567984    5041 scope.go:122] "RemoveContainer" containerID="f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21"
Nov 02 15:45:06.568261 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:06.568238    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2_odh-ai-gateway-infra(a75779b2-f411-4bdc-8b75-4be7f7166687)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687"
Nov 02 15:45:06.650194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:06.650165    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6906-50b046"]
Nov 02 15:45:06.651110 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:06.651084    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb386a213_c422_45ec_a42c_588610440dd7.slice/crio-3d195cb7e9a9134392dbf6faa19f3c566c56d4304292b634f2dae33788a9846f WatchSource:0}: Error finding container 3d195cb7e9a9134392dbf6faa19f3c566c56d4304292b634f2dae33788a9846f: Status 404 returned error can't find the container with id 3d195cb7e9a9134392dbf6faa19f3c566c56d4304292b634f2dae33788a9846f
Nov 02 15:45:06.772921 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:06.772892    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:45:07.542579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:07.542542    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 15:45:07.543089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:07.542599    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 15:45:07.578792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:07.578755    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6906-50b046" event={"ID":"b386a213-c422-45ec-a42c-588610440dd7","Type":"ContainerStarted","Data":"9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6"}
Nov 02 15:45:07.578975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:07.578799    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6906-50b046" event={"ID":"b386a213-c422-45ec-a42c-588610440dd7","Type":"ContainerStarted","Data":"3d195cb7e9a9134392dbf6faa19f3c566c56d4304292b634f2dae33788a9846f"}
Nov 02 15:45:08.542430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.542356    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 15:45:08.542589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.542447    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 15:45:08.711032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.710979    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6906-50b046" podStartSLOduration=3.71096634 podStartE2EDuration="3.71096634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.638516451 +0000 UTC m=+2959.896585902" watchObservedRunningTime="2026-11-02 15:45:08.71096634 +0000 UTC m=+2960.969035789"
Nov 02 15:45:08.730398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.727248    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6906-50b046"]
Nov 02 15:45:08.747036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.747001    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6906-50b046"]
Nov 02 15:45:08.769237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.769205    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6_1a80097a-9d77-41e7-b551-3a71731000ff/payload-processing/1.log"
Nov 02 15:45:08.876394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.876309    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6906-dbc73c"]
Nov 02 15:45:08.877301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.877273    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b386a213-c422-45ec-a42c-588610440dd7" containerName="test-curl-6906-50b046"
Nov 02 15:45:08.877301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.877294    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b386a213-c422-45ec-a42c-588610440dd7" containerName="test-curl-6906-50b046"
Nov 02 15:45:08.877518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.877451    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b386a213-c422-45ec-a42c-588610440dd7" containerName="test-curl-6906-50b046"
Nov 02 15:45:08.881721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.881701    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:08.901398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:08.901341    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6906-dbc73c"]
Nov 02 15:45:09.000095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.000051    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w588f\" (UniqueName: \"kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f\") pod \"test-curl-6906-dbc73c\" (UID: \"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f\") " pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:09.101803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.101762    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w588f\" (UniqueName: \"kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f\") pod \"test-curl-6906-dbc73c\" (UID: \"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f\") " pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:09.138981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.138899    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w588f\" (UniqueName: \"kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f\") pod \"test-curl-6906-dbc73c\" (UID: \"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f\") " pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:09.195895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.195865    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:09.542807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.542325    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6906-dbc73c"]
Nov 02 15:45:09.542807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.542667    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 15:45:09.542807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.542720    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 15:45:09.544327 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:09.544299    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30342c14_80bd_4aeb_aac3_6f3a8eb71a1f.slice/crio-23269fe73a4c4cffd7a4867f9c02808f5bfb463131a9ed60674fa7d1bc78f35c WatchSource:0}: Error finding container 23269fe73a4c4cffd7a4867f9c02808f5bfb463131a9ed60674fa7d1bc78f35c: Status 404 returned error can't find the container with id 23269fe73a4c4cffd7a4867f9c02808f5bfb463131a9ed60674fa7d1bc78f35c
Nov 02 15:45:09.597209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.597175    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6906-dbc73c" event={"ID":"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f","Type":"ContainerStarted","Data":"23269fe73a4c4cffd7a4867f9c02808f5bfb463131a9ed60674fa7d1bc78f35c"}
Nov 02 15:45:09.597469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:09.597445    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6906-50b046" podUID="b386a213-c422-45ec-a42c-588610440dd7" containerName="test-curl-6906-50b046" containerID="cri-o://9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6" gracePeriod=2
Nov 02 15:45:09.608872 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:09.608840    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6906-50b046\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="b386a213-c422-45ec-a42c-588610440dd7" pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:10.541973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:10.541936    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 15:45:10.542585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:10.542545    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 15:45:10.609046 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:10.609004    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6906-dbc73c" event={"ID":"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f","Type":"ContainerStarted","Data":"abe9a950dcdb9125455b4591df946ec170b9f77d6585091fde5e9496b5827c44"}
Nov 02 15:45:10.658078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:10.658028    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6906-dbc73c" podStartSLOduration=2.658015817 podStartE2EDuration="2.658015817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:10.656985664 +0000 UTC m=+2962.915055111" watchObservedRunningTime="2026-11-02 15:45:10.658015817 +0000 UTC m=+2962.916085266"
Nov 02 15:45:11.369272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.369231    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"]
Nov 02 15:45:11.374292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.374271    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.401295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.401261    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"]
Nov 02 15:45:11.474049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474012    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474158    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474203    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6zs\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474229    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474333    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474441    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474495    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474543    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.474599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.474580    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.547181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.547158    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:11.547601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.547579    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:11.548252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.548232    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:45:11.575572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575550    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.575699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575641    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.575699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575676    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6zs\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.575838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575707    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.575838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575724    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.575838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575769    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575853    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575891    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.575962    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.576034    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.576068    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.576381    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.576441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.576560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.576462    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.578268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.578246    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.578546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.578528    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.597404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.597383    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6zs\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.598239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.598214    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.691466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.691432    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:11.855163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.855128    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:11.882856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.882792    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx\" (UniqueName: \"kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx\") pod \"b386a213-c422-45ec-a42c-588610440dd7\" (UID: \"b386a213-c422-45ec-a42c-588610440dd7\") "
Nov 02 15:45:11.884992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.884960    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx" pod "b386a213-c422-45ec-a42c-588610440dd7" (UID: "b386a213-c422-45ec-a42c-588610440dd7"). InnerVolumeSpecName "kube-api-access-gd9bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:11.893657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.893630    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b386a213-c422-45ec-a42c-588610440dd7" path="/var/lib/kubelet/pods/b386a213-c422-45ec-a42c-588610440dd7/volumes"
Nov 02 15:45:11.984599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:11.984563    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9bx\" (UniqueName: \"kubernetes.io/projected/b386a213-c422-45ec-a42c-588610440dd7-kube-api-access-gd9bx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:12.059929 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:12.059902    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4ca564_55bd_49d1_a5ad_4d80b1bfb0c0.slice/crio-53949e4f1dff30e8f8e2217c91fee93db11ec33803602d22f7821f791f36b5c6 WatchSource:0}: Error finding container 53949e4f1dff30e8f8e2217c91fee93db11ec33803602d22f7821f791f36b5c6: Status 404 returned error can't find the container with id 53949e4f1dff30e8f8e2217c91fee93db11ec33803602d22f7821f791f36b5c6
Nov 02 15:45:12.060798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.060771    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"]
Nov 02 15:45:12.062881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.062849    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:12.062966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.062910    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:12.062966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.062939    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:12.630970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.630932    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" event={"ID":"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0","Type":"ContainerStarted","Data":"74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa"}
Nov 02 15:45:12.631440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.630996    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" event={"ID":"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0","Type":"ContainerStarted","Data":"53949e4f1dff30e8f8e2217c91fee93db11ec33803602d22f7821f791f36b5c6"}
Nov 02 15:45:12.632620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.632558    5041 generic.go:356] "Generic (PLEG): container finished" podID="b386a213-c422-45ec-a42c-588610440dd7" containerID="9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6" exitCode=137
Nov 02 15:45:12.632620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.632603    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-50b046"
Nov 02 15:45:12.632620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.632613    5041 scope.go:122] "RemoveContainer" containerID="9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6"
Nov 02 15:45:12.645145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.645128    5041 scope.go:122] "RemoveContainer" containerID="9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6"
Nov 02 15:45:12.645474 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:12.645451    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6\": container with ID starting with 9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6 not found: ID does not exist" containerID="9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6"
Nov 02 15:45:12.645584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.645478    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6"} err="failed to get container status \"9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6\": rpc error: code = NotFound desc = could not find container \"9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6\": container with ID starting with 9124f9b14400b5841ca472a78511a39be2a9fcbd892db678aa2c2834e2e959b6 not found: ID does not exist"
Nov 02 15:45:12.684404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.684344    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podStartSLOduration=1.684331114 podStartE2EDuration="1.684331114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:12.679408801 +0000 UTC m=+2964.937478250" watchObservedRunningTime="2026-11-02 15:45:12.684331114 +0000 UTC m=+2964.942400564"
Nov 02 15:45:12.692485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:12.692460    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:13.694841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:13.694807    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Nov 02 15:45:13.695226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:13.694856    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Nov 02 15:45:14.692809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:14.692775    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Nov 02 15:45:14.692971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:14.692830    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Nov 02 15:45:15.692303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:15.692267    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Nov 02 15:45:15.692734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:15.692323    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Nov 02 15:45:15.880911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:15.880879    5041 scope.go:122] "RemoveContainer" containerID="aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172"
Nov 02 15:45:15.881537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:15.881510    5041 scope.go:122] "RemoveContainer" containerID="073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0"
Nov 02 15:45:16.197020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.196935    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm" podUID="78047769-ae49-463d-9d73-d35afa81b800" containerName="main" containerID="cri-o://abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d" gracePeriod=45
Nov 02 15:45:16.565161 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.564828    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:45:16.674045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.674006    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerStarted","Data":"367a462d2a265cb096b6ba99afdad3775c9f56c08775be949f04006f359e3ec0"}
Nov 02 15:45:16.676792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.676757    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerStarted","Data":"f70215cac29ee90940b7ca98c858050c1a79a647141957d5bc1d182a40be7a7d"}
Nov 02 15:45:16.677597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.677562    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs\" (UniqueName: \"kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs\") pod \"78047769-ae49-463d-9d73-d35afa81b800\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") "
Nov 02 15:45:16.677777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.677642    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir\") pod \"78047769-ae49-463d-9d73-d35afa81b800\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") "
Nov 02 15:45:16.677777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.677760    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home\") pod \"78047769-ae49-463d-9d73-d35afa81b800\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") "
Nov 02 15:45:16.677973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.677792    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp\" (UniqueName: \"kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp\") pod \"78047769-ae49-463d-9d73-d35afa81b800\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") "
Nov 02 15:45:16.677973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.677830    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm\") pod \"78047769-ae49-463d-9d73-d35afa81b800\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") "
Nov 02 15:45:16.677973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.677867    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache\") pod \"78047769-ae49-463d-9d73-d35afa81b800\" (UID: \"78047769-ae49-463d-9d73-d35afa81b800\") "
Nov 02 15:45:16.678956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.678643    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache" pod "78047769-ae49-463d-9d73-d35afa81b800" (UID: "78047769-ae49-463d-9d73-d35afa81b800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:16.678956 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.678662    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home" pod "78047769-ae49-463d-9d73-d35afa81b800" (UID: "78047769-ae49-463d-9d73-d35afa81b800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:16.679633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.679584    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir" pod "78047769-ae49-463d-9d73-d35afa81b800" (UID: "78047769-ae49-463d-9d73-d35afa81b800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:16.680257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.680230    5041 generic.go:356] "Generic (PLEG): container finished" podID="78047769-ae49-463d-9d73-d35afa81b800" containerID="abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d" exitCode=0
Nov 02 15:45:16.680352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.680288    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm" event={"ID":"78047769-ae49-463d-9d73-d35afa81b800","Type":"ContainerDied","Data":"abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d"}
Nov 02 15:45:16.680352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.680318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm" event={"ID":"78047769-ae49-463d-9d73-d35afa81b800","Type":"ContainerDied","Data":"4768b750b44340498670fc6dd7a534783c9df608bb24f9732443bb7c1277c6a9"}
Nov 02 15:45:16.680352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.680343    5041 scope.go:122] "RemoveContainer" containerID="abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d"
Nov 02 15:45:16.680584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.680498    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"
Nov 02 15:45:16.681423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.681355    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp" pod "78047769-ae49-463d-9d73-d35afa81b800" (UID: "78047769-ae49-463d-9d73-d35afa81b800"). InnerVolumeSpecName "kube-api-access-ch7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:16.681578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.681557    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm" pod "78047769-ae49-463d-9d73-d35afa81b800" (UID: "78047769-ae49-463d-9d73-d35afa81b800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:16.681653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.681590    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs" pod "78047769-ae49-463d-9d73-d35afa81b800" (UID: "78047769-ae49-463d-9d73-d35afa81b800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:16.692881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.692856    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Nov 02 15:45:16.693437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.693408    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Nov 02 15:45:16.693572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.693421    5041 scope.go:122] "RemoveContainer" containerID="abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d"
Nov 02 15:45:16.693984 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:16.693952    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d\": container with ID starting with abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d not found: ID does not exist" containerID="abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d"
Nov 02 15:45:16.694081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.693989    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d"} err="failed to get container status \"abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d\": rpc error: code = NotFound desc = could not find container \"abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d\": container with ID starting with abca140535b307406492e02aef09b0ac56200f7b63c5d3141b1f474a082f059d not found: ID does not exist"
Nov 02 15:45:16.779509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.779430    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:16.779509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.779462    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:16.779509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.779473    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78047769-ae49-463d-9d73-d35afa81b800-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:16.779509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.779481    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:16.779509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.779490    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78047769-ae49-463d-9d73-d35afa81b800-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:16.779509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.779498    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7rp\" (UniqueName: \"kubernetes.io/projected/78047769-ae49-463d-9d73-d35afa81b800-kube-api-access-ch7rp\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:16.880520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:16.880491    5041 scope.go:122] "RemoveContainer" containerID="cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630"
Nov 02 15:45:17.085270 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.085230    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"]
Nov 02 15:45:17.147722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.147689    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-453864/auth-test-model-453864-kserve-5b9544ff57-kmlvm"]
Nov 02 15:45:17.692162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.692130    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Nov 02 15:45:17.692330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.692176    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Nov 02 15:45:17.692330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.692217    5041 generic.go:356] "Generic (PLEG): container finished" podID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerID="367a462d2a265cb096b6ba99afdad3775c9f56c08775be949f04006f359e3ec0" exitCode=60
Nov 02 15:45:17.692330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.692312    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerDied","Data":"367a462d2a265cb096b6ba99afdad3775c9f56c08775be949f04006f359e3ec0"}
Nov 02 15:45:17.692510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.692397    5041 scope.go:122] "RemoveContainer" containerID="aae2da72450cce1476fa473b1f0ea94286eccac7e4660c094cbae5af77e4e172"
Nov 02 15:45:17.692791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.692745    5041 scope.go:122] "RemoveContainer" containerID="367a462d2a265cb096b6ba99afdad3775c9f56c08775be949f04006f359e3ec0"
Nov 02 15:45:17.693010 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:17.692992    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc_odh-ai-gateway-infra(0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56"
Nov 02 15:45:17.696344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.696322    5041 generic.go:356] "Generic (PLEG): container finished" podID="5bc27548-b535-4913-b1d3-9debc341f230" containerID="f70215cac29ee90940b7ca98c858050c1a79a647141957d5bc1d182a40be7a7d" exitCode=60
Nov 02 15:45:17.696587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.696572    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerDied","Data":"f70215cac29ee90940b7ca98c858050c1a79a647141957d5bc1d182a40be7a7d"}
Nov 02 15:45:17.696929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.696916    5041 scope.go:122] "RemoveContainer" containerID="f70215cac29ee90940b7ca98c858050c1a79a647141957d5bc1d182a40be7a7d"
Nov 02 15:45:17.697220 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:17.697204    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq_odh-ai-gateway-infra(5bc27548-b535-4913-b1d3-9debc341f230)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" podUID="5bc27548-b535-4913-b1d3-9debc341f230"
Nov 02 15:45:17.700477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.700460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerStarted","Data":"68a2e7f77f5ad72af51e582183a70c2bc4b7237bb93b35adacb24cb21590fa9e"}
Nov 02 15:45:17.707170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.707149    5041 scope.go:122] "RemoveContainer" containerID="073c387fed9f035ef3935c12c2a1f479fc603427cc67fba8039e89cc37f8c9f0"
Nov 02 15:45:17.750288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.750261    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"]
Nov 02 15:45:17.799014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.798985    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"]
Nov 02 15:45:17.894002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:17.893964    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78047769-ae49-463d-9d73-d35afa81b800" path="/var/lib/kubelet/pods/78047769-ae49-463d-9d73-d35afa81b800/volumes"
Nov 02 15:45:18.432859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.432818    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"]
Nov 02 15:45:18.433848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.433823    5041 cpu_manager.go:414] "containerMap: removing container" podUID="78047769-ae49-463d-9d73-d35afa81b800" containerName="main"
Nov 02 15:45:18.433848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.433844    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78047769-ae49-463d-9d73-d35afa81b800" containerName="main"
Nov 02 15:45:18.434084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.434016    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="78047769-ae49-463d-9d73-d35afa81b800" containerName="main"
Nov 02 15:45:18.437041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.437022    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.452354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.452323    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a55a55\""
Nov 02 15:45:18.452538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.452518    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a55a55-dockercfg-lcqn7\""
Nov 02 15:45:18.467992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.467965    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"]
Nov 02 15:45:18.504747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.504642    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrgd\" (UniqueName: \"kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.504747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.504701    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.541301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.541253    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"]
Nov 02 15:45:18.548174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.548142    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.607714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.606354    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.607714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.606704    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrgd\" (UniqueName: \"kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.607714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.606764    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.607714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.606810    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhxl\" (UniqueName: \"kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl\") pod \"payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.610866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.608257    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.630485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.630441    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"]
Nov 02 15:45:18.644389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.643534    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrgd\" (UniqueName: \"kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.707977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.707933    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.708381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.708144    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhxl\" (UniqueName: \"kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl\") pod \"payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.708961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.708934    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.714136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.714100    5041 generic.go:356] "Generic (PLEG): container finished" podID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerID="68a2e7f77f5ad72af51e582183a70c2bc4b7237bb93b35adacb24cb21590fa9e" exitCode=60
Nov 02 15:45:18.714264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.714247    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerDied","Data":"68a2e7f77f5ad72af51e582183a70c2bc4b7237bb93b35adacb24cb21590fa9e"}
Nov 02 15:45:18.714312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.714293    5041 scope.go:122] "RemoveContainer" containerID="cb63cae9e2296bce441357014d8086c49baff5f2ff579a40b7998cd5f3a54630"
Nov 02 15:45:18.714703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.714686    5041 scope.go:122] "RemoveContainer" containerID="68a2e7f77f5ad72af51e582183a70c2bc4b7237bb93b35adacb24cb21590fa9e"
Nov 02 15:45:18.714987 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:18.714970    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn_odh-ai-gateway-infra(5fc7b279-712c-4006-b755-23ed706cfd9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c"
Nov 02 15:45:18.729846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.729816    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhxl\" (UniqueName: \"kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl\") pod \"payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.758896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.755303    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:18.762725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.762694    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:18.763562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.763343    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:18.768402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.766582    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:45:18.864089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.864057    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"]
Nov 02 15:45:18.882221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.881947    5041 scope.go:122] "RemoveContainer" containerID="f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21"
Nov 02 15:45:18.892292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.891958    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:18.961967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:18.960970    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:19.127410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.127358    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2\" (UniqueName: \"kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2\") pod \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") "
Nov 02 15:45:19.127585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.127448    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca\") pod \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\" (UID: \"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56\") "
Nov 02 15:45:19.128105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.128076    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca" pod "0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" (UID: "0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:19.130566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.130408    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2" pod "0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" (UID: "0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56"). InnerVolumeSpecName "kube-api-access-pmnt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:19.141164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.141135    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:19.157164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.157126    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"]
Nov 02 15:45:19.231296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.231256    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd\" (UniqueName: \"kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd\") pod \"5bc27548-b535-4913-b1d3-9debc341f230\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") "
Nov 02 15:45:19.233155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.231558    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca\") pod \"5bc27548-b535-4913-b1d3-9debc341f230\" (UID: \"5bc27548-b535-4913-b1d3-9debc341f230\") "
Nov 02 15:45:19.233155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.232305    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmnt2\" (UniqueName: \"kubernetes.io/projected/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-kube-api-access-pmnt2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:19.233155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.232329    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:19.233155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.232534    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca" pod "5bc27548-b535-4913-b1d3-9debc341f230" (UID: "5bc27548-b535-4913-b1d3-9debc341f230"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:19.240536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.240503    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd" pod "5bc27548-b535-4913-b1d3-9debc341f230" (UID: "5bc27548-b535-4913-b1d3-9debc341f230"). InnerVolumeSpecName "kube-api-access-sktkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:19.242145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.242122    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"]
Nov 02 15:45:19.243564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.243539    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:19.243706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.243693    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:19.243875 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.243863    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.243955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.243944    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.244054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244044    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.244124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244114    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.244217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244208    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:19.244290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244280    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:19.244381 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244356    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.244455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244445    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.244768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244752    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.244870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244860    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:19.244952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.244943    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.245034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.245025    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" containerName="cleanup"
Nov 02 15:45:19.245117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.245107    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:19.249288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.249267    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.271332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.271300    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"]
Nov 02 15:45:19.298446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.298023    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"]
Nov 02 15:45:19.333904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.333882    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.333991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.333933    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334095    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334120    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334149    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334300    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334404    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334443    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvn6r\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334474    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.334711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334693    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktkd\" (UniqueName: \"kubernetes.io/projected/5bc27548-b535-4913-b1d3-9debc341f230-kube-api-access-sktkd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:19.334782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.334718    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc27548-b535-4913-b1d3-9debc341f230-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:19.436320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436289    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436348    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436472    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436491    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436517    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436546    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436587    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436625    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvn6r\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436654    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436873    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.436918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.436906    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.437307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.437271    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.437484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.437461    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.437916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.437893    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.438488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.438468    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.438766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.438750    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.457636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.457608    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.457785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.457767    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvn6r\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.593593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.593530    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:19.730982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.730951    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"
Nov 02 15:45:19.731485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.730980    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc" event={"ID":"0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56","Type":"ContainerDied","Data":"9033e3ccf19e8c1fb6a46ded8e648bec7f10194b3477908d980973005ade8006"}
Nov 02 15:45:19.731485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.731057    5041 scope.go:122] "RemoveContainer" containerID="367a462d2a265cb096b6ba99afdad3775c9f56c08775be949f04006f359e3ec0"
Nov 02 15:45:19.732686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.732665    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"
Nov 02 15:45:19.732820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.732715    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq" event={"ID":"5bc27548-b535-4913-b1d3-9debc341f230","Type":"ContainerDied","Data":"ea5899c9093c5d85cf6d491de041a981928b14a7d0e7a918aed5691a505df3de"}
Nov 02 15:45:19.735141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.735111    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerStarted","Data":"0e82dd9ca0d58dab178ee26919b4dbf24845f40bb0eea7920c7b639ca641b2b0"}
Nov 02 15:45:19.737214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.737180    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7" event={"ID":"47da5dd3-8a24-41b1-a29d-cb8accb18eff","Type":"ContainerStarted","Data":"2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03"}
Nov 02 15:45:19.737321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.737229    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7" event={"ID":"47da5dd3-8a24-41b1-a29d-cb8accb18eff","Type":"ContainerStarted","Data":"392e9030efe11a69a8a060e17b6489f79fac60d5c9d2970d7f3c4deae9d56c28"}
Nov 02 15:45:19.737477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.737457    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:19.741423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.741164    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj" event={"ID":"65e47cda-7a73-44a1-9646-d75526f9c07c","Type":"ContainerStarted","Data":"1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8"}
Nov 02 15:45:19.741423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.741212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj" event={"ID":"65e47cda-7a73-44a1-9646-d75526f9c07c","Type":"ContainerStarted","Data":"f9c333ee7a7a2f56a2f6cc1d5524cd475be8edaf4a720dfd9a233fb93b52994b"}
Nov 02 15:45:19.742382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.741629    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:19.754942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.754922    5041 scope.go:122] "RemoveContainer" containerID="f70215cac29ee90940b7ca98c858050c1a79a647141957d5bc1d182a40be7a7d"
Nov 02 15:45:19.786786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.786727    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7" podStartSLOduration=1.786708722 podStartE2EDuration="1.786708722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:19.778931713 +0000 UTC m=+2972.037001163" watchObservedRunningTime="2026-11-02 15:45:19.786708722 +0000 UTC m=+2972.044778174"
Nov 02 15:45:19.870517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.867651    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj" podStartSLOduration=1.867632723 podStartE2EDuration="1.867632723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:19.857530595 +0000 UTC m=+2972.115600046" watchObservedRunningTime="2026-11-02 15:45:19.867632723 +0000 UTC m=+2972.125702174"
Nov 02 15:45:19.911561 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.911534    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"]
Nov 02 15:45:19.914043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.914015    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:19.932538 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.929760    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893905-dxtxq"]
Nov 02 15:45:19.949103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.949072    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"]
Nov 02 15:45:19.963635 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:19.963604    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5effd6d_da48_41cb_a71b_7590b17e4e7d.slice/crio-fc027719734356b3bde14661fd5047df68e52796f580b06bd707a5d133f33692 WatchSource:0}: Error finding container fc027719734356b3bde14661fd5047df68e52796f580b06bd707a5d133f33692: Status 404 returned error can't find the container with id fc027719734356b3bde14661fd5047df68e52796f580b06bd707a5d133f33692
Nov 02 15:45:19.968720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.968684    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:19.968850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.968762    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:19.968850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:19.968806    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:20.007172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.007148    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"]
Nov 02 15:45:20.012287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.012265    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f84ad8-29893905-zbzcc"]
Nov 02 15:45:20.050495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.050472    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm\" (UniqueName: \"kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm\") pod \"5fc7b279-712c-4006-b755-23ed706cfd9c\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") "
Nov 02 15:45:20.050703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.050680    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca\") pod \"5fc7b279-712c-4006-b755-23ed706cfd9c\" (UID: \"5fc7b279-712c-4006-b755-23ed706cfd9c\") "
Nov 02 15:45:20.051100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.051077    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca" pod "5fc7b279-712c-4006-b755-23ed706cfd9c" (UID: "5fc7b279-712c-4006-b755-23ed706cfd9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:20.053398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.052470    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc7b279-712c-4006-b755-23ed706cfd9c-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:20.053398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.053256    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm" pod "5fc7b279-712c-4006-b755-23ed706cfd9c" (UID: "5fc7b279-712c-4006-b755-23ed706cfd9c"). InnerVolumeSpecName "kube-api-access-r7nmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:20.153581 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.153489    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nmm\" (UniqueName: \"kubernetes.io/projected/5fc7b279-712c-4006-b755-23ed706cfd9c-kube-api-access-r7nmm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:20.755266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.755237    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"
Nov 02 15:45:20.755730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.755242    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn" event={"ID":"5fc7b279-712c-4006-b755-23ed706cfd9c","Type":"ContainerDied","Data":"fc8ba48b46fd2c22aa5955d7cadfc19bcc0316429171828ccbb819c37d916d11"}
Nov 02 15:45:20.755730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.755405    5041 scope.go:122] "RemoveContainer" containerID="68a2e7f77f5ad72af51e582183a70c2bc4b7237bb93b35adacb24cb21590fa9e"
Nov 02 15:45:20.759745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.759716    5041 generic.go:356] "Generic (PLEG): container finished" podID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerID="0e82dd9ca0d58dab178ee26919b4dbf24845f40bb0eea7920c7b639ca641b2b0" exitCode=60
Nov 02 15:45:20.759891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.759787    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerDied","Data":"0e82dd9ca0d58dab178ee26919b4dbf24845f40bb0eea7920c7b639ca641b2b0"}
Nov 02 15:45:20.760200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.760183    5041 scope.go:122] "RemoveContainer" containerID="0e82dd9ca0d58dab178ee26919b4dbf24845f40bb0eea7920c7b639ca641b2b0"
Nov 02 15:45:20.760595 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:20.760555    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2_odh-ai-gateway-infra(a75779b2-f411-4bdc-8b75-4be7f7166687)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687"
Nov 02 15:45:20.761982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.761955    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" event={"ID":"a5effd6d-da48-41cb-a71b-7590b17e4e7d","Type":"ContainerStarted","Data":"0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a"}
Nov 02 15:45:20.762108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.762007    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" event={"ID":"a5effd6d-da48-41cb-a71b-7590b17e4e7d","Type":"ContainerStarted","Data":"fc027719734356b3bde14661fd5047df68e52796f580b06bd707a5d133f33692"}
Nov 02 15:45:20.769030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.769014    5041 scope.go:122] "RemoveContainer" containerID="f95c08967b12ff84ef5e012312550c0f5aa7ecb03775e9f024dd1775eb92af21"
Nov 02 15:45:20.830158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.830134    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"]
Nov 02 15:45:20.847407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.847044    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podStartSLOduration=1.847029063 podStartE2EDuration="1.847029063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:20.844567184 +0000 UTC m=+2973.102636634" watchObservedRunningTime="2026-11-02 15:45:20.847029063 +0000 UTC m=+2973.105098513"
Nov 02 15:45:20.881235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.881200    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"]
Nov 02 15:45:20.886742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:20.886719    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893905-jzprn"]
Nov 02 15:45:21.594496 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.594464    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:21.596003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.595974    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 15:45:21.596156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.596025    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 15:45:21.647456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.647421    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6906-dbc73c"]
Nov 02 15:45:21.647715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.647689    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6906-dbc73c" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" containerName="test-curl-6906-dbc73c" containerID="cri-o://abe9a950dcdb9125455b4591df946ec170b9f77d6585091fde5e9496b5827c44" gracePeriod=2
Nov 02 15:45:21.655834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.655452    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6906-dbc73c"]
Nov 02 15:45:21.658940 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:21.658905    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6906-dbc73c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:21.783949 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:21.783911    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6906-dbc73c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:21.898889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.898853    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56" path="/var/lib/kubelet/pods/0cdefad5-aff3-4ae9-bda7-0f30dc5c4f56/volumes"
Nov 02 15:45:21.899618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.899590    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc27548-b535-4913-b1d3-9debc341f230" path="/var/lib/kubelet/pods/5bc27548-b535-4913-b1d3-9debc341f230/volumes"
Nov 02 15:45:21.900220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.900199    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" path="/var/lib/kubelet/pods/5fc7b279-712c-4006-b755-23ed706cfd9c/volumes"
Nov 02 15:45:21.995658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:21.995632    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:22.085348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.085308    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r\" (UniqueName: \"kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r\") pod \"a75779b2-f411-4bdc-8b75-4be7f7166687\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") "
Nov 02 15:45:22.085563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.085431    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca\") pod \"a75779b2-f411-4bdc-8b75-4be7f7166687\" (UID: \"a75779b2-f411-4bdc-8b75-4be7f7166687\") "
Nov 02 15:45:22.085963 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.085930    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca" pod "a75779b2-f411-4bdc-8b75-4be7f7166687" (UID: "a75779b2-f411-4bdc-8b75-4be7f7166687"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:22.086474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.086244    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75779b2-f411-4bdc-8b75-4be7f7166687-maas-api-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:22.087916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.087885    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r" pod "a75779b2-f411-4bdc-8b75-4be7f7166687" (UID: "a75779b2-f411-4bdc-8b75-4be7f7166687"). InnerVolumeSpecName "kube-api-access-gv92r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:22.187909 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.187878    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv92r\" (UniqueName: \"kubernetes.io/projected/a75779b2-f411-4bdc-8b75-4be7f7166687-kube-api-access-gv92r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:22.594150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.594051    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 15:45:22.594150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.594123    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 15:45:22.784052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.784009    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2" event={"ID":"a75779b2-f411-4bdc-8b75-4be7f7166687","Type":"ContainerDied","Data":"56317efc37ddf201b76d36fe2794393a259224e17e7b88fd2ce15c4d8300b6cb"}
Nov 02 15:45:22.784574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.784090    5041 scope.go:122] "RemoveContainer" containerID="0e82dd9ca0d58dab178ee26919b4dbf24845f40bb0eea7920c7b639ca641b2b0"
Nov 02 15:45:22.784574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.784032    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"
Nov 02 15:45:22.845121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.845052    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"]
Nov 02 15:45:22.853842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:22.853815    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6f2e0c-29893905-jmvs2"]
Nov 02 15:45:23.594756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:23.594719    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 15:45:23.594910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:23.594783    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 15:45:23.806533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:23.805563    5041 generic.go:356] "Generic (PLEG): container finished" podID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" containerID="abe9a950dcdb9125455b4591df946ec170b9f77d6585091fde5e9496b5827c44" exitCode=137
Nov 02 15:45:23.895852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:23.895765    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" path="/var/lib/kubelet/pods/a75779b2-f411-4bdc-8b75-4be7f7166687/volumes"
Nov 02 15:45:23.964560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:23.964523    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:24.062803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.062770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f\" (UniqueName: \"kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f\") pod \"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f\" (UID: \"30342c14-80bd-4aeb-aac3-6f3a8eb71a1f\") "
Nov 02 15:45:24.065053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.065022    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f" pod "30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" (UID: "30342c14-80bd-4aeb-aac3-6f3a8eb71a1f"). InnerVolumeSpecName "kube-api-access-w588f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:24.164837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.164763    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w588f\" (UniqueName: \"kubernetes.io/projected/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f-kube-api-access-w588f\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.594926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.594886    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 15:45:24.595116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.594951    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 15:45:24.845971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.845893    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:24.845971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:24.845947    5041 scope.go:122] "RemoveContainer" containerID="abe9a950dcdb9125455b4591df946ec170b9f77d6585091fde5e9496b5827c44"
Nov 02 15:45:24.866972 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:24.866938    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6906-dbc73c\" is forbidden: User \"system:node:ip-10-0-133-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-143.ec2.internal' and this object" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" pod="opendatahub/test-curl-6906-dbc73c"
Nov 02 15:45:25.593847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:25.593804    5041 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 15:45:25.594054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:25.593861    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 15:45:25.889991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:25.889915    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" path="/var/lib/kubelet/pods/30342c14-80bd-4aeb-aac3-6f3a8eb71a1f/volumes"
Nov 02 15:45:26.655958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:26.655919    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:26.656805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:26.656766    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:26.657908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:26.657888    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:45:30.765020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:30.764985    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:45:30.765492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:30.765087    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:45:33.629849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.629691    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"]
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631107    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631136    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631168    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631177    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631206    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631213    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631239    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631245    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631254    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631260    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631281    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631288    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631307    5041 cpu_manager.go:414] "containerMap: removing container" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" containerName="test-curl-6906-dbc73c"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631313    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" containerName="test-curl-6906-dbc73c"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631580    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631599    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc27548-b535-4913-b1d3-9debc341f230" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631616    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631629    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631642    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.635414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.631650    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="30342c14-80bd-4aeb-aac3-6f3a8eb71a1f" containerName="test-curl-6906-dbc73c"
Nov 02 15:45:33.636644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.636383    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.664178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.664140    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dd139\""
Nov 02 15:45:33.664356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.664274    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dd139-dockercfg-84g8n\""
Nov 02 15:45:33.668147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.668105    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"]
Nov 02 15:45:33.697292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.697262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjpt\" (UniqueName: \"kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.697439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.697337    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.799174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.799137    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.799469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.799434    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjpt\" (UniqueName: \"kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.799987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.799962    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.804992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.804968    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"]
Nov 02 15:45:33.805918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.805900    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.805981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.805919    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.806096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.806083    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc7b279-712c-4006-b755-23ed706cfd9c" containerName="cleanup"
Nov 02 15:45:33.806152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.806104    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75779b2-f411-4bdc-8b75-4be7f7166687" containerName="cleanup"
Nov 02 15:45:33.809432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.809412    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:33.825106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.825086    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjpt\" (UniqueName: \"kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt\") pod \"payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:33.830359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.830337    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"]
Nov 02 15:45:33.900886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.900814    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:33.901024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.901009    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q268v\" (UniqueName: \"kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v\") pod \"payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:33.954258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:33.954228    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:34.002659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.002628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q268v\" (UniqueName: \"kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v\") pod \"payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:34.002823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.002806    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:34.003806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.003776    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:34.025248 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.024460    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q268v\" (UniqueName: \"kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v\") pod \"payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:34.124745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.124706    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:34.274344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.274307    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"]
Nov 02 15:45:34.286240 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:34.286201    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dd7387_31f6_4a7f_892c_1b188b98eb4f.slice/crio-d5fa6b1612e2a990fde0d59644758eb8ef74f065cbbb6cba5a6a19680cdc1223 WatchSource:0}: Error finding container d5fa6b1612e2a990fde0d59644758eb8ef74f065cbbb6cba5a6a19680cdc1223: Status 404 returned error can't find the container with id d5fa6b1612e2a990fde0d59644758eb8ef74f065cbbb6cba5a6a19680cdc1223
Nov 02 15:45:34.436122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.436043    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"]
Nov 02 15:45:34.440304 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:34.440266    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37222fad_eaa6_4a52_a010_37f4c2c41d96.slice/crio-41a4e24295845061081d21d313b47fd3f58d7cd85021f9340f9fadca1cd3bc4b WatchSource:0}: Error finding container 41a4e24295845061081d21d313b47fd3f58d7cd85021f9340f9fadca1cd3bc4b: Status 404 returned error can't find the container with id 41a4e24295845061081d21d313b47fd3f58d7cd85021f9340f9fadca1cd3bc4b
Nov 02 15:45:34.957377 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.957323    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc" event={"ID":"37222fad-eaa6-4a52-a010-37f4c2c41d96","Type":"ContainerStarted","Data":"b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26"}
Nov 02 15:45:34.957850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.957418    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc" event={"ID":"37222fad-eaa6-4a52-a010-37f4c2c41d96","Type":"ContainerStarted","Data":"41a4e24295845061081d21d313b47fd3f58d7cd85021f9340f9fadca1cd3bc4b"}
Nov 02 15:45:34.957850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.957455    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:34.959468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.959435    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v" event={"ID":"99dd7387-31f6-4a7f-892c-1b188b98eb4f","Type":"ContainerStarted","Data":"df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db"}
Nov 02 15:45:34.959606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.959488    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v" event={"ID":"99dd7387-31f6-4a7f-892c-1b188b98eb4f","Type":"ContainerStarted","Data":"d5fa6b1612e2a990fde0d59644758eb8ef74f065cbbb6cba5a6a19680cdc1223"}
Nov 02 15:45:34.959606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.959560    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:34.995456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:34.995415    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc" podStartSLOduration=1.995401489 podStartE2EDuration="1.995401489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:34.993746971 +0000 UTC m=+2987.251816421" watchObservedRunningTime="2026-11-02 15:45:34.995401489 +0000 UTC m=+2987.253470939"
Nov 02 15:45:35.031311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:35.031272    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v" podStartSLOduration=2.0312567440000002 podStartE2EDuration="2.031256744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:35.028994422 +0000 UTC m=+2987.287063872" watchObservedRunningTime="2026-11-02 15:45:35.031256744 +0000 UTC m=+2987.289326194"
Nov 02 15:45:36.771404 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:36.770747    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:45:40.637710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:40.637671    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"]
Nov 02 15:45:40.638252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:40.637895    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" podUID="4174a84d-f201-40ff-8310-dfc750701f87" containerName="maas-api" containerID="cri-o://a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51" gracePeriod=30
Nov 02 15:45:40.793519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:40.793473    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" podUID="4174a84d-f201-40ff-8310-dfc750701f87" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.8:8443/health\": dial tcp 10.132.1.8:8443: connect: connection refused"
Nov 02 15:45:40.979711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:40.979687    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:45:41.018194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.018160    5041 generic.go:356] "Generic (PLEG): container finished" podID="4174a84d-f201-40ff-8310-dfc750701f87" containerID="a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51" exitCode=0
Nov 02 15:45:41.018352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.018210    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"
Nov 02 15:45:41.018352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.018239    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" event={"ID":"4174a84d-f201-40ff-8310-dfc750701f87","Type":"ContainerDied","Data":"a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51"}
Nov 02 15:45:41.018352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.018294    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd" event={"ID":"4174a84d-f201-40ff-8310-dfc750701f87","Type":"ContainerDied","Data":"5832b27895bf5fc0aa0a1444d4f508321e5d76d7b182efece279d3b60337fa09"}
Nov 02 15:45:41.018352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.018325    5041 scope.go:122] "RemoveContainer" containerID="a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51"
Nov 02 15:45:41.033120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.033096    5041 scope.go:122] "RemoveContainer" containerID="a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51"
Nov 02 15:45:41.033421 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:45:41.033398    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51\": container with ID starting with a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51 not found: ID does not exist" containerID="a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51"
Nov 02 15:45:41.033507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.033427    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51"} err="failed to get container status \"a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51\": rpc error: code = NotFound desc = could not find container \"a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51\": container with ID starting with a4c7ccce588269f3718a56ec02af7b474de514c8590ca664b89a450299335e51 not found: ID does not exist"
Nov 02 15:45:41.099251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.099219    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm\" (UniqueName: \"kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm\") pod \"4174a84d-f201-40ff-8310-dfc750701f87\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") "
Nov 02 15:45:41.099414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.099257    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls\") pod \"4174a84d-f201-40ff-8310-dfc750701f87\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") "
Nov 02 15:45:41.099414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.099403    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca\") pod \"4174a84d-f201-40ff-8310-dfc750701f87\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") "
Nov 02 15:45:41.099555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.099446    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs\") pod \"4174a84d-f201-40ff-8310-dfc750701f87\" (UID: \"4174a84d-f201-40ff-8310-dfc750701f87\") "
Nov 02 15:45:41.099807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.099779    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca" pod "4174a84d-f201-40ff-8310-dfc750701f87" (UID: "4174a84d-f201-40ff-8310-dfc750701f87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:41.100134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.100096    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4174a84d-f201-40ff-8310-dfc750701f87-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.101359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.101332    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls" pod "4174a84d-f201-40ff-8310-dfc750701f87" (UID: "4174a84d-f201-40ff-8310-dfc750701f87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:41.101601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.101577    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs" pod "4174a84d-f201-40ff-8310-dfc750701f87" (UID: "4174a84d-f201-40ff-8310-dfc750701f87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:41.101648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.101620    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm" pod "4174a84d-f201-40ff-8310-dfc750701f87" (UID: "4174a84d-f201-40ff-8310-dfc750701f87"). InnerVolumeSpecName "kube-api-access-5tjkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:41.245474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.237608    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tjkm\" (UniqueName: \"kubernetes.io/projected/4174a84d-f201-40ff-8310-dfc750701f87-kube-api-access-5tjkm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.245474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.237697    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.245474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.237713    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4174a84d-f201-40ff-8310-dfc750701f87-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.366643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.366612    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"]
Nov 02 15:45:41.377772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.377741    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9cfd1f-74df798685-wvjnd"]
Nov 02 15:45:41.890239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:41.890207    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4174a84d-f201-40ff-8310-dfc750701f87" path="/var/lib/kubelet/pods/4174a84d-f201-40ff-8310-dfc750701f87/volumes"
Nov 02 15:45:45.974054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:45.974022    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:45:45.974497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:45.974096    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:45:49.261502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:49.261347    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:45:49.271255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:49.271234    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:45:55.879751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.879701    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"]
Nov 02 15:45:55.880834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.880807    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4174a84d-f201-40ff-8310-dfc750701f87" containerName="maas-api"
Nov 02 15:45:55.880834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.880833    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4174a84d-f201-40ff-8310-dfc750701f87" containerName="maas-api"
Nov 02 15:45:55.881104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.881089    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4174a84d-f201-40ff-8310-dfc750701f87" containerName="maas-api"
Nov 02 15:45:55.887264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.887237    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:55.901025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.900995    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-24255b-dockercfg-rkm95\""
Nov 02 15:45:55.904553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.904529    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"]
Nov 02 15:45:55.908292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:55.908271    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-24255b\""
Nov 02 15:45:56.001868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.001834    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"]
Nov 02 15:45:56.068487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.068448    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.068745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.068720    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9tm\" (UniqueName: \"kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm\") pod \"payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.069321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.069291    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.093810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.093783    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"]
Nov 02 15:45:56.169948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.169923    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.170115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.169992    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.170115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.170046    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9tm\" (UniqueName: \"kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm\") pod \"payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.170253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.170162    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjvs\" (UniqueName: \"kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs\") pod \"payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.170759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.170735    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.219900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.219870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9tm\" (UniqueName: \"kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm\") pod \"payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.271755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.271720    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjvs\" (UniqueName: \"kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs\") pod \"payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.271969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.271950    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.272831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.272802    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.292680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.292659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjvs\" (UniqueName: \"kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs\") pod \"payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.394933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.394904    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:56.503258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.503231    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:56.706287 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:56.706254    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba55d13c_e299_4508_8ca2_3918533ac4af.slice/crio-2449df4c033371defaf8b81b55a81a4d39f7b117a4c9a2a7ca023247bb52267e WatchSource:0}: Error finding container 2449df4c033371defaf8b81b55a81a4d39f7b117a4c9a2a7ca023247bb52267e: Status 404 returned error can't find the container with id 2449df4c033371defaf8b81b55a81a4d39f7b117a4c9a2a7ca023247bb52267e
Nov 02 15:45:56.706673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.706619    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"]
Nov 02 15:45:56.824316 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:45:56.824286    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2059821_9d01_4178_a072_ce18f726ec18.slice/crio-7974a6d2b4c8c3366cc4721f3d4a7a864acab5002c47a2a8f3721cd1902ad5e6 WatchSource:0}: Error finding container 7974a6d2b4c8c3366cc4721f3d4a7a864acab5002c47a2a8f3721cd1902ad5e6: Status 404 returned error can't find the container with id 7974a6d2b4c8c3366cc4721f3d4a7a864acab5002c47a2a8f3721cd1902ad5e6
Nov 02 15:45:56.824443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:56.824395    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"]
Nov 02 15:45:57.173516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.173479    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7" event={"ID":"ba55d13c-e299-4508-8ca2-3918533ac4af","Type":"ContainerStarted","Data":"1d7ff6c03c81319781ac9eafd986fde6bff08edd106c777b27b246a87e123765"}
Nov 02 15:45:57.173977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.173566    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7" event={"ID":"ba55d13c-e299-4508-8ca2-3918533ac4af","Type":"ContainerStarted","Data":"2449df4c033371defaf8b81b55a81a4d39f7b117a4c9a2a7ca023247bb52267e"}
Nov 02 15:45:57.173977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.173697    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:45:57.175407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.175378    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt" event={"ID":"d2059821-9d01-4178-a072-ce18f726ec18","Type":"ContainerStarted","Data":"565264c00dd87bdb56be40b59e9a19cc8c11868a908b41dfbdf099a4711a0d5b"}
Nov 02 15:45:57.175535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.175428    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt" event={"ID":"d2059821-9d01-4178-a072-ce18f726ec18","Type":"ContainerStarted","Data":"7974a6d2b4c8c3366cc4721f3d4a7a864acab5002c47a2a8f3721cd1902ad5e6"}
Nov 02 15:45:57.175578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.175544    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:45:57.211511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.211239    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7" podStartSLOduration=2.211226991 podStartE2EDuration="2.211226991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:57.208510555 +0000 UTC m=+3009.466580006" watchObservedRunningTime="2026-11-02 15:45:57.211226991 +0000 UTC m=+3009.469296441"
Nov 02 15:45:57.244066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:57.244028    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt" podStartSLOduration=2.244017241 podStartE2EDuration="2.244017241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:57.243556999 +0000 UTC m=+3009.501626449" watchObservedRunningTime="2026-11-02 15:45:57.244017241 +0000 UTC m=+3009.502086691"
Nov 02 15:45:58.003583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.003548    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"]
Nov 02 15:45:58.003825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.003800    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" podUID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" containerName="maas-api" containerID="cri-o://84cc4bc5daf825805fcd81c066a47b8e77f7f144a605cfc9c1fae2d2500d2d50" gracePeriod=30
Nov 02 15:45:58.186835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.186693    5041 generic.go:356] "Generic (PLEG): container finished" podID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" containerID="84cc4bc5daf825805fcd81c066a47b8e77f7f144a605cfc9c1fae2d2500d2d50" exitCode=0
Nov 02 15:45:58.186835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.186780    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" event={"ID":"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4","Type":"ContainerDied","Data":"84cc4bc5daf825805fcd81c066a47b8e77f7f144a605cfc9c1fae2d2500d2d50"}
Nov 02 15:45:58.319675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.319652    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:45:58.509693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.509652    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5\" (UniqueName: \"kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5\") pod \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") "
Nov 02 15:45:58.509693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.509696    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs\") pod \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") "
Nov 02 15:45:58.509948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.509742    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls\") pod \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") "
Nov 02 15:45:58.510017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.509953    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca\") pod \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\" (UID: \"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4\") "
Nov 02 15:45:58.510436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.510315    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca" pod "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" (UID: "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:58.510686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.510665    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.511953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.511928    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5" pod "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" (UID: "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4"). InnerVolumeSpecName "kube-api-access-cj4k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:58.512048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.511946    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls" pod "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" (UID: "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:58.512048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.511930    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs" pod "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" (UID: "54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:58.612593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.612558    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj4k5\" (UniqueName: \"kubernetes.io/projected/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-kube-api-access-cj4k5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.612593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.612590    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.612593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:58.612600    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:45:59.197837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:59.197807    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"
Nov 02 15:45:59.198294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:59.197844    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" event={"ID":"54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4","Type":"ContainerDied","Data":"c09592c72295a1078e05508a3672e13ad8f4c0bc442d4b05d3d8e93e607fefa3"}
Nov 02 15:45:59.198294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:59.197907    5041 scope.go:122] "RemoveContainer" containerID="84cc4bc5daf825805fcd81c066a47b8e77f7f144a605cfc9c1fae2d2500d2d50"
Nov 02 15:45:59.245949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:59.245921    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"]
Nov 02 15:45:59.258571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:59.258541    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh"]
Nov 02 15:45:59.892788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:45:59.892752    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" path="/var/lib/kubelet/pods/54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4/volumes"
Nov 02 15:46:01.302228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:01.302186    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f84ad8-56745589fc-6rdrh" podUID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.11:8443/health\": context deadline exceeded"
Nov 02 15:46:02.402942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.402907    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"]
Nov 02 15:46:02.403801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.403785    5041 cpu_manager.go:414] "containerMap: removing container" podUID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" containerName="maas-api"
Nov 02 15:46:02.403801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.403801    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" containerName="maas-api"
Nov 02 15:46:02.404031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.404021    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c2573c-a19c-42c7-ac42-2ebc0b3cc5a4" containerName="maas-api"
Nov 02 15:46:02.410110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.410089    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.425709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.425683    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a55a55-serving-cert\""
Nov 02 15:46:02.433334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.432869    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"]
Nov 02 15:46:02.472222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.472182    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.472468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.472236    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjc7m\" (UniqueName: \"kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.472468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.472298    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.472468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.472321    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.575268    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.575322    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjc7m\" (UniqueName: \"kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.575405    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.575429    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:02.575635    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-serving-cert: secret "maas-api-e2e-ipp-a-a55a55-serving-cert" not found
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:02.575717    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls podName:58285986-2b9a-4804-9494-5c61afe725d7 nodeName:}" failed. No retries permitted until 2026-11-02 15:46:03.075695692 +0000 UTC m=+3015.333765140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls") pod "maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" (UID: "58285986-2b9a-4804-9494-5c61afe725d7") : secret "maas-api-e2e-ipp-a-a55a55-serving-cert" not found
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.576161    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.584422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.579888    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:02.599959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:02.599933    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjc7m\" (UniqueName: \"kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:03.083518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:03.083468    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:03.086420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:03.086395    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:03.330049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:03.330012    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:03.676507 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:03.675711    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"]
Nov 02 15:46:03.701306 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:46:03.701271    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58285986_2b9a_4804_9494_5c61afe725d7.slice/crio-14e61fe3fe178af7e4ffa60b1c6b3280411afc32b78c3790f85b86889d868c92 WatchSource:0}: Error finding container 14e61fe3fe178af7e4ffa60b1c6b3280411afc32b78c3790f85b86889d868c92: Status 404 returned error can't find the container with id 14e61fe3fe178af7e4ffa60b1c6b3280411afc32b78c3790f85b86889d868c92
Nov 02 15:46:04.261940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.261911    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"]
Nov 02 15:46:04.262474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.262135    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st" podUID="248d0912-8f3f-4198-a3b7-1596e6da6dec" containerName="maas-api" containerID="cri-o://cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3" gracePeriod=30
Nov 02 15:46:04.264686 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.264656    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" event={"ID":"58285986-2b9a-4804-9494-5c61afe725d7","Type":"ContainerStarted","Data":"14e61fe3fe178af7e4ffa60b1c6b3280411afc32b78c3790f85b86889d868c92"}
Nov 02 15:46:04.607656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.607627    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:46:04.711211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.711122    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls\") pod \"248d0912-8f3f-4198-a3b7-1596e6da6dec\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") "
Nov 02 15:46:04.711704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.711301    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc\" (UniqueName: \"kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc\") pod \"248d0912-8f3f-4198-a3b7-1596e6da6dec\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") "
Nov 02 15:46:04.711704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.711548    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs\") pod \"248d0912-8f3f-4198-a3b7-1596e6da6dec\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") "
Nov 02 15:46:04.711704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.711608    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca\") pod \"248d0912-8f3f-4198-a3b7-1596e6da6dec\" (UID: \"248d0912-8f3f-4198-a3b7-1596e6da6dec\") "
Nov 02 15:46:04.718883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.712621    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca" pod "248d0912-8f3f-4198-a3b7-1596e6da6dec" (UID: "248d0912-8f3f-4198-a3b7-1596e6da6dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:04.718883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.714031    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls" pod "248d0912-8f3f-4198-a3b7-1596e6da6dec" (UID: "248d0912-8f3f-4198-a3b7-1596e6da6dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:04.718883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.714720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc" pod "248d0912-8f3f-4198-a3b7-1596e6da6dec" (UID: "248d0912-8f3f-4198-a3b7-1596e6da6dec"). InnerVolumeSpecName "kube-api-access-nwzdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:04.718883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.714811    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs" pod "248d0912-8f3f-4198-a3b7-1596e6da6dec" (UID: "248d0912-8f3f-4198-a3b7-1596e6da6dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:04.827387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.813714    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzdc\" (UniqueName: \"kubernetes.io/projected/248d0912-8f3f-4198-a3b7-1596e6da6dec-kube-api-access-nwzdc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:04.827387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.813766    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:04.827387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.813793    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248d0912-8f3f-4198-a3b7-1596e6da6dec-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:04.827387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:04.813813    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248d0912-8f3f-4198-a3b7-1596e6da6dec-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.277555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.276534    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" event={"ID":"58285986-2b9a-4804-9494-5c61afe725d7","Type":"ContainerStarted","Data":"e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323"}
Nov 02 15:46:05.277555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.277541    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:05.295112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.295073    5041 generic.go:356] "Generic (PLEG): container finished" podID="248d0912-8f3f-4198-a3b7-1596e6da6dec" containerID="cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3" exitCode=0
Nov 02 15:46:05.295357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.295142    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"
Nov 02 15:46:05.295357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.295152    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st" event={"ID":"248d0912-8f3f-4198-a3b7-1596e6da6dec","Type":"ContainerDied","Data":"cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3"}
Nov 02 15:46:05.295357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.295286    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st" event={"ID":"248d0912-8f3f-4198-a3b7-1596e6da6dec","Type":"ContainerDied","Data":"71253ee7b7a4ea5fee532762f3c94b2ce97a102f96e1b31e1bcde4d7dd653a15"}
Nov 02 15:46:05.295357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.295315    5041 scope.go:122] "RemoveContainer" containerID="cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3"
Nov 02 15:46:05.314143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.314118    5041 scope.go:122] "RemoveContainer" containerID="cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3"
Nov 02 15:46:05.314545 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:05.314519    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3\": container with ID starting with cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3 not found: ID does not exist" containerID="cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3"
Nov 02 15:46:05.314623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.314550    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3"} err="failed to get container status \"cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3\": rpc error: code = NotFound desc = could not find container \"cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3\": container with ID starting with cb45e601f0ea6b574b07c89ba1e90530ad1027019c9990ddaf83cbd1b8a818c3 not found: ID does not exist"
Nov 02 15:46:05.325111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.325058    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" podStartSLOduration=2.854667359 podStartE2EDuration="3.325042706s" totalImagesPullingTime="470.375347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:05.319084146 +0000 UTC m=+3017.577153595" watchObservedRunningTime="2026-11-02 15:46:05.325042706 +0000 UTC m=+3017.583112156"
Nov 02 15:46:05.354532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.354509    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"]
Nov 02 15:46:05.361392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.361351    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3805b8-6688b4b8c7-pv4st"]
Nov 02 15:46:05.834891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.834856    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"]
Nov 02 15:46:05.894823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:05.894557    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248d0912-8f3f-4198-a3b7-1596e6da6dec" path="/var/lib/kubelet/pods/248d0912-8f3f-4198-a3b7-1596e6da6dec/volumes"
Nov 02 15:46:06.770592 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:06.770556    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:46:08.190666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:08.190634    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:46:08.191239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:08.191077    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:46:11.314554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:11.314524    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:46:12.338752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:12.338719    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"]
Nov 02 15:46:12.339241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:12.338971    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy" containerID="cri-o://ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938" gracePeriod=30
Nov 02 15:46:12.350314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:12.350284    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"]
Nov 02 15:46:12.350702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:12.350671    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy" containerID="cri-o://b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928" gracePeriod=30
Nov 02 15:46:12.368539 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:12.368507    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"]
Nov 02 15:46:12.368777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:12.368748    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy" containerID="cri-o://0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a" gracePeriod=30
Nov 02 15:46:17.706479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.706445    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:46:17.774782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.774747    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:46:17.781514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.781489    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:46:17.819968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.819933    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.819993    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820065    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820143    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820335    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820431    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820482    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820571    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.820666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.820610    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token\") pod \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\" (UID: \"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a\") "
Nov 02 15:46:17.821620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.821590    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.821903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.821879    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.822096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.822071    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.822273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.822252    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.822684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.822658    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.825622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.825464    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.825622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.825584    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.826043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.826016    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.838103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.838080    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q" pod "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" (UID: "880ceacb-6c92-4c7a-b0ef-7c2d72b2697a"). InnerVolumeSpecName "kube-api-access-pr72q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.922498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922468    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.922651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922525    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.922651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922581    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.922651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922601    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.922827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922736    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.922827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922794    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.922947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922896    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.923016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922959    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.923016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.922990    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.923118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923016    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.923118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923045    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.923118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923111    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.923276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923183    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.923276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923218    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.923276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923269    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.923475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923303    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.923475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923354    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.923475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923424    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs\") pod \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\" (UID: \"a5effd6d-da48-41cb-a71b-7590b17e4e7d\") "
Nov 02 15:46:17.923475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.923462    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy\") pod \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\" (UID: \"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425\") "
Nov 02 15:46:17.924341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924308    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924341    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924356    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924385    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924399    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr72q\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-kube-api-access-pr72q\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924412    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924426    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924439    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924452    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.924578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924472    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924560    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924641    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924731    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.924947    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.925004    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.925095    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.925110    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.925128    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.925442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.925271    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.926211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.926187    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.926857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.926838    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "kube-api-access-rf8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.927225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.927198    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.927293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.927268    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.927465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.927444    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.928447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.928424    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.934779 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.934747    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r" pod "a5effd6d-da48-41cb-a71b-7590b17e4e7d" (UID: "a5effd6d-da48-41cb-a71b-7590b17e4e7d"). InnerVolumeSpecName "kube-api-access-zvn6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.948680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:17.948656    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token" pod "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" (UID: "9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:18.026307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026275    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026301    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026310    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf8jg\" (UniqueName: \"kubernetes.io/projected/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-kube-api-access-rf8jg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026320    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvn6r\" (UniqueName: \"kubernetes.io/projected/a5effd6d-da48-41cb-a71b-7590b17e4e7d-kube-api-access-zvn6r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026330    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026339    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026353    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026377    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026387    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026396    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026404    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026411    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026423    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026432    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026440    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026449    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5effd6d-da48-41cb-a71b-7590b17e4e7d-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.026483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.026457    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.462487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.462455    5041 generic.go:356] "Generic (PLEG): container finished" podID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerID="0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a" exitCode=0
Nov 02 15:46:18.462621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.462500    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"
Nov 02 15:46:18.462621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.462544    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" event={"ID":"a5effd6d-da48-41cb-a71b-7590b17e4e7d","Type":"ContainerDied","Data":"0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a"}
Nov 02 15:46:18.462621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.462612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97" event={"ID":"a5effd6d-da48-41cb-a71b-7590b17e4e7d","Type":"ContainerDied","Data":"fc027719734356b3bde14661fd5047df68e52796f580b06bd707a5d133f33692"}
Nov 02 15:46:18.462834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.462645    5041 scope.go:122] "RemoveContainer" containerID="0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a"
Nov 02 15:46:18.464191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.464158    5041 generic.go:356] "Generic (PLEG): container finished" podID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerID="ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938" exitCode=0
Nov 02 15:46:18.464301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.464192    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" event={"ID":"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425","Type":"ContainerDied","Data":"ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938"}
Nov 02 15:46:18.464301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.464250    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf" event={"ID":"9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425","Type":"ContainerDied","Data":"aec3352f0c5faa26befe4b51e8fb0ff304213e95e429c59966978d31f8322205"}
Nov 02 15:46:18.464301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.464200    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"
Nov 02 15:46:18.465954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.465928    5041 generic.go:356] "Generic (PLEG): container finished" podID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerID="b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928" exitCode=0
Nov 02 15:46:18.466071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.465956    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" event={"ID":"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a","Type":"ContainerDied","Data":"b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928"}
Nov 02 15:46:18.466071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.465982    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"
Nov 02 15:46:18.466071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.465986    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw" event={"ID":"880ceacb-6c92-4c7a-b0ef-7c2d72b2697a","Type":"ContainerDied","Data":"e5b416abb6174d975707fd43e741a911033b9f97457c280727c0b1c4639a23a6"}
Nov 02 15:46:18.476881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.476710    5041 scope.go:122] "RemoveContainer" containerID="0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a"
Nov 02 15:46:18.476979 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:18.476963    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a\": container with ID starting with 0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a not found: ID does not exist" containerID="0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a"
Nov 02 15:46:18.477021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.476984    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a"} err="failed to get container status \"0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a\": rpc error: code = NotFound desc = could not find container \"0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a\": container with ID starting with 0406ddaecd1844576cc7f7164dbf3b37d39f298f1f8a89289c8e8100679a114a not found: ID does not exist"
Nov 02 15:46:18.477021 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.477000    5041 scope.go:122] "RemoveContainer" containerID="ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938"
Nov 02 15:46:18.503791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.503767    5041 scope.go:122] "RemoveContainer" containerID="ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938"
Nov 02 15:46:18.504106 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:18.504084    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938\": container with ID starting with ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938 not found: ID does not exist" containerID="ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938"
Nov 02 15:46:18.504155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.504111    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938"} err="failed to get container status \"ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938\": rpc error: code = NotFound desc = could not find container \"ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938\": container with ID starting with ee7eb832f897372c9a7915f0785627666f3194ca45a8db2a88124f3530b82938 not found: ID does not exist"
Nov 02 15:46:18.504155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.504127    5041 scope.go:122] "RemoveContainer" containerID="b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928"
Nov 02 15:46:18.511563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.511540    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"]
Nov 02 15:46:18.515745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.515715    5041 scope.go:122] "RemoveContainer" containerID="b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928"
Nov 02 15:46:18.515972 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:18.515951    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928\": container with ID starting with b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928 not found: ID does not exist" containerID="b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928"
Nov 02 15:46:18.516052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.515978    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928"} err="failed to get container status \"b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928\": rpc error: code = NotFound desc = could not find container \"b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928\": container with ID starting with b5319da50d05de616b6c37b510e57bd61ea092759a6676bcd1d4e5cca9865928 not found: ID does not exist"
Nov 02 15:46:18.517327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.517305    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sxpkw"]
Nov 02 15:46:18.547542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.547520    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"]
Nov 02 15:46:18.553471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.553450    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-7mp97"]
Nov 02 15:46:18.580957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.580933    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"]
Nov 02 15:46:18.585225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:18.585201    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-g68mf"]
Nov 02 15:46:19.890607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:19.890576    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" path="/var/lib/kubelet/pods/880ceacb-6c92-4c7a-b0ef-7c2d72b2697a/volumes"
Nov 02 15:46:19.891071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:19.891056    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" path="/var/lib/kubelet/pods/9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425/volumes"
Nov 02 15:46:19.891521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:19.891507    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" path="/var/lib/kubelet/pods/a5effd6d-da48-41cb-a71b-7590b17e4e7d/volumes"
Nov 02 15:46:20.872233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:20.872195    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh" podUID="2055d9e5-cd3f-4fcb-8ba9-ba5439793983" containerName="main" containerID="cri-o://44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4" gracePeriod=45
Nov 02 15:46:21.159764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.159737    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:46:21.266691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.266659    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm\") pod \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") "
Nov 02 15:46:21.266890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.266848    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs\" (UniqueName: \"kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs\") pod \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") "
Nov 02 15:46:21.266992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.266972    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache\") pod \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") "
Nov 02 15:46:21.267075 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.267039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir\") pod \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") "
Nov 02 15:46:21.267137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.267086    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns\" (UniqueName: \"kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns\") pod \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") "
Nov 02 15:46:21.267207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.267186    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home\") pod \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\" (UID: \"2055d9e5-cd3f-4fcb-8ba9-ba5439793983\") "
Nov 02 15:46:21.267651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.267621    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home" pod "2055d9e5-cd3f-4fcb-8ba9-ba5439793983" (UID: "2055d9e5-cd3f-4fcb-8ba9-ba5439793983"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:21.267651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.267639    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir" pod "2055d9e5-cd3f-4fcb-8ba9-ba5439793983" (UID: "2055d9e5-cd3f-4fcb-8ba9-ba5439793983"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:21.268030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.268006    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:21.268128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.268036    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:21.268189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.268148    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache" pod "2055d9e5-cd3f-4fcb-8ba9-ba5439793983" (UID: "2055d9e5-cd3f-4fcb-8ba9-ba5439793983"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:21.269174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.269154    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm" pod "2055d9e5-cd3f-4fcb-8ba9-ba5439793983" (UID: "2055d9e5-cd3f-4fcb-8ba9-ba5439793983"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:21.269354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.269325    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs" pod "2055d9e5-cd3f-4fcb-8ba9-ba5439793983" (UID: "2055d9e5-cd3f-4fcb-8ba9-ba5439793983"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:21.269689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.269667    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns" pod "2055d9e5-cd3f-4fcb-8ba9-ba5439793983" (UID: "2055d9e5-cd3f-4fcb-8ba9-ba5439793983"). InnerVolumeSpecName "kube-api-access-58pns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:21.369813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.369781    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:21.369813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.369809    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58pns\" (UniqueName: \"kubernetes.io/projected/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-kube-api-access-58pns\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:21.369813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.369821    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:21.370092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.369831    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2055d9e5-cd3f-4fcb-8ba9-ba5439793983-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:21.497273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.497244    5041 generic.go:356] "Generic (PLEG): container finished" podID="2055d9e5-cd3f-4fcb-8ba9-ba5439793983" containerID="44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4" exitCode=0
Nov 02 15:46:21.497468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.497283    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"
Nov 02 15:46:21.497468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.497337    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh" event={"ID":"2055d9e5-cd3f-4fcb-8ba9-ba5439793983","Type":"ContainerDied","Data":"44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4"}
Nov 02 15:46:21.497468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.497413    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh" event={"ID":"2055d9e5-cd3f-4fcb-8ba9-ba5439793983","Type":"ContainerDied","Data":"17c5a01c0578d8a71f9e9a4bcdaee36f0b473b8d8e80c1bda2a6c9a18d3faf23"}
Nov 02 15:46:21.497468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.497437    5041 scope.go:122] "RemoveContainer" containerID="44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4"
Nov 02 15:46:21.509769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.509752    5041 scope.go:122] "RemoveContainer" containerID="44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4"
Nov 02 15:46:21.510022 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:21.510002    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4\": container with ID starting with 44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4 not found: ID does not exist" containerID="44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4"
Nov 02 15:46:21.510143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.510028    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4"} err="failed to get container status \"44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4\": rpc error: code = NotFound desc = could not find container \"44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4\": container with ID starting with 44bd1f893409e891c0d2c16d97b08ff3424acc2f6d89d172a6d15f0c88d86da4 not found: ID does not exist"
Nov 02 15:46:21.543941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.543918    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"]
Nov 02 15:46:21.550056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.550036    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce791/sub-test-model-cce791-kserve-7f84d5c9d9-fhpwh"]
Nov 02 15:46:21.892028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:21.891949    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2055d9e5-cd3f-4fcb-8ba9-ba5439793983" path="/var/lib/kubelet/pods/2055d9e5-cd3f-4fcb-8ba9-ba5439793983/volumes"
Nov 02 15:46:24.664407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.664349    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"]
Nov 02 15:46:24.665302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665280    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy"
Nov 02 15:46:24.665302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665300    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665314    5041 cpu_manager.go:414] "containerMap: removing container" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665320    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665352    5041 cpu_manager.go:414] "containerMap: removing container" podUID="248d0912-8f3f-4198-a3b7-1596e6da6dec" containerName="maas-api"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665357    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248d0912-8f3f-4198-a3b7-1596e6da6dec" containerName="maas-api"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665384    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2055d9e5-cd3f-4fcb-8ba9-ba5439793983" containerName="main"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665390    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2055d9e5-cd3f-4fcb-8ba9-ba5439793983" containerName="main"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665402    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy"
Nov 02 15:46:24.665493 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665407    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy"
Nov 02 15:46:24.665773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665535    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="248d0912-8f3f-4198-a3b7-1596e6da6dec" containerName="maas-api"
Nov 02 15:46:24.665773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665546    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2055d9e5-cd3f-4fcb-8ba9-ba5439793983" containerName="main"
Nov 02 15:46:24.665773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665556    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5effd6d-da48-41cb-a71b-7590b17e4e7d" containerName="istio-proxy"
Nov 02 15:46:24.665773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665567    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e080a3f-5ef7-4c1f-8a1b-d14f6f1f1425" containerName="istio-proxy"
Nov 02 15:46:24.665773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.665575    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="880ceacb-6c92-4c7a-b0ef-7c2d72b2697a" containerName="istio-proxy"
Nov 02 15:46:24.671202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.671182    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.689124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.689102    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6f2e0c-openshift-default-dockercfg-mpw2f\""
Nov 02 15:46:24.709877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.709847    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"]
Nov 02 15:46:24.812039 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812008    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812057    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mq2\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812130    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812198 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812188    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812216    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812318    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812483    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812548    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.812606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.812585    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.914324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914286    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.914608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914350    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mq2\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.914608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914416    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.914608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.914608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914513    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.914608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914568    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914673    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914750    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.914791    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.915122    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.915246    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.915682    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.915981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.915956    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.916871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.916843    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.917552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.917503    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.928497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.919348    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.941877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.941847    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mq2\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.942089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.942066    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:24.986908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:24.986882    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:25.319539 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:46:25.319506    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07664b84_0e99_4d03_ae3f_a3ea5518be3f.slice/crio-0b3adb8fe9ddb1b40996b22b34f8b7e9f78c2abd1bd719f5fc12c17dbb467160 WatchSource:0}: Error finding container 0b3adb8fe9ddb1b40996b22b34f8b7e9f78c2abd1bd719f5fc12c17dbb467160: Status 404 returned error can't find the container with id 0b3adb8fe9ddb1b40996b22b34f8b7e9f78c2abd1bd719f5fc12c17dbb467160
Nov 02 15:46:25.320420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.320392    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"]
Nov 02 15:46:25.322354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.322308    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:25.322488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.322399    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:25.322488 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.322432    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:25.538017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.537979    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" event={"ID":"07664b84-0e99-4d03-ae3f-a3ea5518be3f","Type":"ContainerStarted","Data":"1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2"}
Nov 02 15:46:25.538200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.538040    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" event={"ID":"07664b84-0e99-4d03-ae3f-a3ea5518be3f","Type":"ContainerStarted","Data":"0b3adb8fe9ddb1b40996b22b34f8b7e9f78c2abd1bd719f5fc12c17dbb467160"}
Nov 02 15:46:25.582437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.582298    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podStartSLOduration=1.582279968 podStartE2EDuration="1.582279968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:25.580667184 +0000 UTC m=+3037.838736633" watchObservedRunningTime="2026-11-02 15:46:25.582279968 +0000 UTC m=+3037.840349411"
Nov 02 15:46:25.987824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:25.987792    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:26.989103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:26.989067    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Nov 02 15:46:26.989530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:26.989121    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Nov 02 15:46:27.987847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:27.987812    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Nov 02 15:46:27.988030 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:27.987862    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Nov 02 15:46:28.987582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:28.987548    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Nov 02 15:46:28.988008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:28.987602    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Nov 02 15:46:29.988312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:29.987836    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Nov 02 15:46:29.989113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:29.988309    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Nov 02 15:46:30.987639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:30.987603    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Nov 02 15:46:30.987858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:30.987661    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Nov 02 15:46:32.173079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:32.173040    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:32.174126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:32.174100    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:32.175178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:32.175155    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:46:36.769596 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:36.769558    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:46:36.835800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:36.835771    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"]
Nov 02 15:46:36.843344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:36.843323    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:36.855774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:36.855739    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce791-openshift-default-dockercfg-w7mhr\""
Nov 02 15:46:36.871125 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:36.871103    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"]
Nov 02 15:46:37.006256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.006218    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.006461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.006381    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nxz\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.006600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.006580    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.006685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.006669    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.006746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.006727    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.006822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.006760    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.007068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.007046    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.007153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.007135    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.007284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.007259    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.108988    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109037    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109086    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109113    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nxz\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109205    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109250    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109315    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109449    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109566    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109696    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.109797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.109762    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.110081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.110065    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.111618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.111593    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.111836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.111817    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.130440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.130415    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.130638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.130618    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nxz\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.159807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.159782    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:37.486066 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:46:37.486034    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58e3d37_fc86_46dd_8cb4_fab66bca973d.slice/crio-0e2b407b4117638df2eab1e3dfeb904b8670743230417cc3330b8f77b6fc57e7 WatchSource:0}: Error finding container 0e2b407b4117638df2eab1e3dfeb904b8670743230417cc3330b8f77b6fc57e7: Status 404 returned error can't find the container with id 0e2b407b4117638df2eab1e3dfeb904b8670743230417cc3330b8f77b6fc57e7
Nov 02 15:46:37.486413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.486387    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"]
Nov 02 15:46:37.488894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.488864    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:37.488964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.488927    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:37.488964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.488957    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:37.653064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.653021    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" event={"ID":"c58e3d37-fc86-46dd-8cb4-fab66bca973d","Type":"ContainerStarted","Data":"8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3"}
Nov 02 15:46:37.653242 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.653100    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" event={"ID":"c58e3d37-fc86-46dd-8cb4-fab66bca973d","Type":"ContainerStarted","Data":"0e2b407b4117638df2eab1e3dfeb904b8670743230417cc3330b8f77b6fc57e7"}
Nov 02 15:46:37.693308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:37.693248    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podStartSLOduration=1.6932300580000001 podStartE2EDuration="1.693230058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:37.692428505 +0000 UTC m=+3049.950497951" watchObservedRunningTime="2026-11-02 15:46:37.693230058 +0000 UTC m=+3049.951299510"
Nov 02 15:46:38.160469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:38.160436    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:38.162455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:38.162424    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 15:46:38.162606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:38.162472    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 15:46:39.160594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:39.160558    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 15:46:39.160990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:39.160618    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 15:46:40.161016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:40.160933    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 15:46:40.161016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:40.160989    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 15:46:41.160266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:41.160229    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 15:46:41.160474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:41.160288    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 15:46:42.160868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:42.160835    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 15:46:42.161265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:42.160888    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 15:46:43.161156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:43.161121    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 15:46:43.161592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:43.161176    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 15:46:44.165683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:44.165649    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:44.166096 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:44.166035    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:44.166594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:44.166572    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:46:52.831464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:52.831425    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"]
Nov 02 15:46:52.864232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:52.864204    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"]
Nov 02 15:46:57.831901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:57.831837    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj" podUID="65e47cda-7a73-44a1-9646-d75526f9c07c" containerName="payload-processing" containerID="cri-o://1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8" gracePeriod=25
Nov 02 15:46:57.842607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:57.842573    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"]
Nov 02 15:46:57.842843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:57.842813    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn" podUID="d334f924-047f-4f60-8ca1-6d8751dd9833" containerName="maas-api" containerID="cri-o://ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d" gracePeriod=30
Nov 02 15:46:57.873645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:57.873606    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7" podUID="47da5dd3-8a24-41b1-a29d-cb8accb18eff" containerName="payload-pre-processing" containerID="cri-o://2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03" gracePeriod=25
Nov 02 15:46:58.008911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.008871    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"]
Nov 02 15:46:58.009549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.009288    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing" containerID="cri-o://df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea" gracePeriod=30
Nov 02 15:46:58.243087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.242669    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:46:58.277702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.277675    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:46:58.287045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.287013    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume\") pod \"65e47cda-7a73-44a1-9646-d75526f9c07c\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") "
Nov 02 15:46:58.287195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.287073    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl\" (UniqueName: \"kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl\") pod \"65e47cda-7a73-44a1-9646-d75526f9c07c\" (UID: \"65e47cda-7a73-44a1-9646-d75526f9c07c\") "
Nov 02 15:46:58.293414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.293381    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume" pod "65e47cda-7a73-44a1-9646-d75526f9c07c" (UID: "65e47cda-7a73-44a1-9646-d75526f9c07c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:58.296627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.296601    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl" pod "65e47cda-7a73-44a1-9646-d75526f9c07c" (UID: "65e47cda-7a73-44a1-9646-d75526f9c07c"). InnerVolumeSpecName "kube-api-access-pmhxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:58.296769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.296689    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:46:58.396962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.396862    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls\") pod \"d334f924-047f-4f60-8ca1-6d8751dd9833\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") "
Nov 02 15:46:58.397735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.397708    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs\") pod \"d334f924-047f-4f60-8ca1-6d8751dd9833\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") "
Nov 02 15:46:58.398001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.397986    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume\") pod \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") "
Nov 02 15:46:58.398158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.398144    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca\") pod \"d334f924-047f-4f60-8ca1-6d8751dd9833\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") "
Nov 02 15:46:58.398345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.398330    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf\" (UniqueName: \"kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf\") pod \"d334f924-047f-4f60-8ca1-6d8751dd9833\" (UID: \"d334f924-047f-4f60-8ca1-6d8751dd9833\") "
Nov 02 15:46:58.400283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.400258    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd\" (UniqueName: \"kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd\") pod \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\" (UID: \"47da5dd3-8a24-41b1-a29d-cb8accb18eff\") "
Nov 02 15:46:58.402613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.401208    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65e47cda-7a73-44a1-9646-d75526f9c07c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.402613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.401237    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhxl\" (UniqueName: \"kubernetes.io/projected/65e47cda-7a73-44a1-9646-d75526f9c07c-kube-api-access-pmhxl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.402613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.398662    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume" pod "47da5dd3-8a24-41b1-a29d-cb8accb18eff" (UID: "47da5dd3-8a24-41b1-a29d-cb8accb18eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:58.402613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.399077    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca" pod "d334f924-047f-4f60-8ca1-6d8751dd9833" (UID: "d334f924-047f-4f60-8ca1-6d8751dd9833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:58.405106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.405053    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls" pod "d334f924-047f-4f60-8ca1-6d8751dd9833" (UID: "d334f924-047f-4f60-8ca1-6d8751dd9833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:58.434463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.432843    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf" pod "d334f924-047f-4f60-8ca1-6d8751dd9833" (UID: "d334f924-047f-4f60-8ca1-6d8751dd9833"). InnerVolumeSpecName "kube-api-access-pgcpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:58.434463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.432952    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd" pod "47da5dd3-8a24-41b1-a29d-cb8accb18eff" (UID: "47da5dd3-8a24-41b1-a29d-cb8accb18eff"). InnerVolumeSpecName "kube-api-access-2xrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:58.434814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.434774    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs" pod "d334f924-047f-4f60-8ca1-6d8751dd9833" (UID: "d334f924-047f-4f60-8ca1-6d8751dd9833"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:58.483409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.481673    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:46:58.503425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.503246    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47da5dd3-8a24-41b1-a29d-cb8accb18eff-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.503425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.503290    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d334f924-047f-4f60-8ca1-6d8751dd9833-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.503425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.503309    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgcpf\" (UniqueName: \"kubernetes.io/projected/d334f924-047f-4f60-8ca1-6d8751dd9833-kube-api-access-pgcpf\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.503425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.503321    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xrgd\" (UniqueName: \"kubernetes.io/projected/47da5dd3-8a24-41b1-a29d-cb8accb18eff-kube-api-access-2xrgd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.503425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.503337    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.503425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.503350    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d334f924-047f-4f60-8ca1-6d8751dd9833-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.607634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.605620    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume\") pod \"1a80097a-9d77-41e7-b551-3a71731000ff\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") "
Nov 02 15:46:58.607634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.605958    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k\" (UniqueName: \"kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k\") pod \"1a80097a-9d77-41e7-b551-3a71731000ff\" (UID: \"1a80097a-9d77-41e7-b551-3a71731000ff\") "
Nov 02 15:46:58.609495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.608053    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume" pod "1a80097a-9d77-41e7-b551-3a71731000ff" (UID: "1a80097a-9d77-41e7-b551-3a71731000ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:58.642584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.642532    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k" pod "1a80097a-9d77-41e7-b551-3a71731000ff" (UID: "1a80097a-9d77-41e7-b551-3a71731000ff"). InnerVolumeSpecName "kube-api-access-f987k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:58.708265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.708229    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f987k\" (UniqueName: \"kubernetes.io/projected/1a80097a-9d77-41e7-b551-3a71731000ff-kube-api-access-f987k\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.708265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.708263    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a80097a-9d77-41e7-b551-3a71731000ff-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.869352    5041 generic.go:356] "Generic (PLEG): container finished" podID="47da5dd3-8a24-41b1-a29d-cb8accb18eff" containerID="2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03" exitCode=0
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.869492    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.869503    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7" event={"ID":"47da5dd3-8a24-41b1-a29d-cb8accb18eff","Type":"ContainerDied","Data":"2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03"}
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.869603    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7" event={"ID":"47da5dd3-8a24-41b1-a29d-cb8accb18eff","Type":"ContainerDied","Data":"392e9030efe11a69a8a060e17b6489f79fac60d5c9d2970d7f3c4deae9d56c28"}
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.869625    5041 scope.go:122] "RemoveContainer" containerID="2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03"
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.871572    5041 generic.go:356] "Generic (PLEG): container finished" podID="65e47cda-7a73-44a1-9646-d75526f9c07c" containerID="1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8" exitCode=0
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.871791    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj" event={"ID":"65e47cda-7a73-44a1-9646-d75526f9c07c","Type":"ContainerDied","Data":"1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8"}
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.871837    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj" event={"ID":"65e47cda-7a73-44a1-9646-d75526f9c07c","Type":"ContainerDied","Data":"f9c333ee7a7a2f56a2f6cc1d5524cd475be8edaf4a720dfd9a233fb93b52994b"}
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.871935    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.876995    5041 generic.go:356] "Generic (PLEG): container finished" podID="d334f924-047f-4f60-8ca1-6d8751dd9833" containerID="ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d" exitCode=0
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.877202    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.879694    5041 generic.go:356] "Generic (PLEG): container finished" podID="1a80097a-9d77-41e7-b551-3a71731000ff" containerID="df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea" exitCode=0
Nov 02 15:46:58.886036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.879830    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"
Nov 02 15:46:58.887166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.886896    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn" event={"ID":"d334f924-047f-4f60-8ca1-6d8751dd9833","Type":"ContainerDied","Data":"ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d"}
Nov 02 15:46:58.887166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.886940    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn" event={"ID":"d334f924-047f-4f60-8ca1-6d8751dd9833","Type":"ContainerDied","Data":"584c3bbb2ea3ff7646e515c28ee5ad33686eb2a590743006a95b53acad911cda"}
Nov 02 15:46:58.887166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.886956    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" event={"ID":"1a80097a-9d77-41e7-b551-3a71731000ff","Type":"ContainerDied","Data":"df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea"}
Nov 02 15:46:58.887166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.886975    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6" event={"ID":"1a80097a-9d77-41e7-b551-3a71731000ff","Type":"ContainerDied","Data":"c7229ef9d2d3343edde2ef3d57244d92a8351ced79e816ce922a49c506e1da2a"}
Nov 02 15:46:58.897500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.897482    5041 scope.go:122] "RemoveContainer" containerID="2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03"
Nov 02 15:46:58.897758 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:58.897741    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03\": container with ID starting with 2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03 not found: ID does not exist" containerID="2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03"
Nov 02 15:46:58.897836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.897765    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03"} err="failed to get container status \"2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03\": rpc error: code = NotFound desc = could not find container \"2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03\": container with ID starting with 2f01495578630fef36f19f1e84d9492e651757b2a088c96446ed7b776fa3bb03 not found: ID does not exist"
Nov 02 15:46:58.897836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.897782    5041 scope.go:122] "RemoveContainer" containerID="1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8"
Nov 02 15:46:58.908517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.908495    5041 scope.go:122] "RemoveContainer" containerID="1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8"
Nov 02 15:46:58.908767 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:58.908747    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8\": container with ID starting with 1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8 not found: ID does not exist" containerID="1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8"
Nov 02 15:46:58.908821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.908773    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8"} err="failed to get container status \"1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8\": rpc error: code = NotFound desc = could not find container \"1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8\": container with ID starting with 1810e5c56a78d88cddecd60acce0eaacf1a1dd1965e2e12171821e77b92bffa8 not found: ID does not exist"
Nov 02 15:46:58.908821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.908789    5041 scope.go:122] "RemoveContainer" containerID="ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d"
Nov 02 15:46:58.918635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.918619    5041 scope.go:122] "RemoveContainer" containerID="ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d"
Nov 02 15:46:58.918871 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:58.918853    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d\": container with ID starting with ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d not found: ID does not exist" containerID="ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d"
Nov 02 15:46:58.918922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.918878    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d"} err="failed to get container status \"ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d\": rpc error: code = NotFound desc = could not find container \"ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d\": container with ID starting with ae2dcb90c6e9f03eb7e7254a6e54c5885f11079d270800a162585a10eaa8721d not found: ID does not exist"
Nov 02 15:46:58.918922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.918893    5041 scope.go:122] "RemoveContainer" containerID="df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea"
Nov 02 15:46:58.928823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.928797    5041 scope.go:122] "RemoveContainer" containerID="2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"
Nov 02 15:46:58.938636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.938619    5041 scope.go:122] "RemoveContainer" containerID="df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea"
Nov 02 15:46:58.938901 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:58.938881    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea\": container with ID starting with df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea not found: ID does not exist" containerID="df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea"
Nov 02 15:46:58.938957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.938908    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea"} err="failed to get container status \"df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea\": rpc error: code = NotFound desc = could not find container \"df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea\": container with ID starting with df41a287cff5d85f1bcbd548afbeeea7045d3f921cb994f215e3710b76d1ceea not found: ID does not exist"
Nov 02 15:46:58.938957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.938923    5041 scope.go:122] "RemoveContainer" containerID="2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"
Nov 02 15:46:58.939162 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:46:58.939144    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f\": container with ID starting with 2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f not found: ID does not exist" containerID="2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"
Nov 02 15:46:58.939209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.939168    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f"} err="failed to get container status \"2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f\": rpc error: code = NotFound desc = could not find container \"2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f\": container with ID starting with 2652e22cb9ea12f880abb6852fa08d76d634160dd9fa435748c7516a3553b01f not found: ID does not exist"
Nov 02 15:46:58.960147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.960088    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"]
Nov 02 15:46:58.964863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.964836    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-7c78457fbf-v5zrj"]
Nov 02 15:46:58.993111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.993089    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"]
Nov 02 15:46:58.997574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:58.997549    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e80c77-84b4c84f8f-wz5qn"]
Nov 02 15:46:59.026143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.026117    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"]
Nov 02 15:46:59.030336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.030314    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-679bb58c5b-747w6"]
Nov 02 15:46:59.057809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.057779    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"]
Nov 02 15:46:59.062011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.061989    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-6b87784787-m5bp7"]
Nov 02 15:46:59.891200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.891167    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" path="/var/lib/kubelet/pods/1a80097a-9d77-41e7-b551-3a71731000ff/volumes"
Nov 02 15:46:59.891753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.891735    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47da5dd3-8a24-41b1-a29d-cb8accb18eff" path="/var/lib/kubelet/pods/47da5dd3-8a24-41b1-a29d-cb8accb18eff/volumes"
Nov 02 15:46:59.892285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.892264    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e47cda-7a73-44a1-9646-d75526f9c07c" path="/var/lib/kubelet/pods/65e47cda-7a73-44a1-9646-d75526f9c07c/volumes"
Nov 02 15:46:59.893121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:46:59.893098    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d334f924-047f-4f60-8ca1-6d8751dd9833" path="/var/lib/kubelet/pods/d334f924-047f-4f60-8ca1-6d8751dd9833/volumes"
Nov 02 15:47:06.373831 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:06.373785    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"]
Nov 02 15:47:06.374345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:06.374039    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy" containerID="cri-o://a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7" gracePeriod=30
Nov 02 15:47:06.409055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:06.409019    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"]
Nov 02 15:47:06.409238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:06.409218    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy" containerID="cri-o://a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc" gracePeriod=30
Nov 02 15:47:06.413588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:06.413563    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"]
Nov 02 15:47:06.413763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:06.413744    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy" containerID="cri-o://74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa" gracePeriod=30
Nov 02 15:47:06.770173 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:06.770137    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:47:07.621701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.621665    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"]
Nov 02 15:47:07.622213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.621922    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" podUID="63bff263-0915-44d7-92e1-781e193fdf12" containerName="maas-api" containerID="cri-o://4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c" gracePeriod=30
Nov 02 15:47:07.922511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.922480    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:47:07.972062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.972029    5041 generic.go:356] "Generic (PLEG): container finished" podID="63bff263-0915-44d7-92e1-781e193fdf12" containerID="4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c" exitCode=0
Nov 02 15:47:07.972265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.972087    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"
Nov 02 15:47:07.972265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.972113    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" event={"ID":"63bff263-0915-44d7-92e1-781e193fdf12","Type":"ContainerDied","Data":"4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c"}
Nov 02 15:47:07.972265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.972177    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq" event={"ID":"63bff263-0915-44d7-92e1-781e193fdf12","Type":"ContainerDied","Data":"b9ba5962a96d21d77779a1ea0c080ce73e91e2d0b693836f929e5cbbfede8bca"}
Nov 02 15:47:07.972265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.972207    5041 scope.go:122] "RemoveContainer" containerID="4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c"
Nov 02 15:47:07.985684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.985658    5041 scope.go:122] "RemoveContainer" containerID="4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c"
Nov 02 15:47:07.985960 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:07.985935    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c\": container with ID starting with 4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c not found: ID does not exist" containerID="4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c"
Nov 02 15:47:07.986060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:07.985966    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c"} err="failed to get container status \"4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c\": rpc error: code = NotFound desc = could not find container \"4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c\": container with ID starting with 4901703564c4a19d1fd8c82d9f17ee781fe3a9e3bc33bcf7b5c38d118b569f9c not found: ID does not exist"
Nov 02 15:47:08.039651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.039621    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs\") pod \"63bff263-0915-44d7-92e1-781e193fdf12\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") "
Nov 02 15:47:08.039802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.039746    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\") pod \"63bff263-0915-44d7-92e1-781e193fdf12\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") "
Nov 02 15:47:08.039802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.039796    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn\") pod \"63bff263-0915-44d7-92e1-781e193fdf12\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") "
Nov 02 15:47:08.039883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.039870    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca\") pod \"63bff263-0915-44d7-92e1-781e193fdf12\" (UID: \"63bff263-0915-44d7-92e1-781e193fdf12\") "
Nov 02 15:47:08.040305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.040277    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca" pod "63bff263-0915-44d7-92e1-781e193fdf12" (UID: "63bff263-0915-44d7-92e1-781e193fdf12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:08.040569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.040545    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bff263-0915-44d7-92e1-781e193fdf12-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:08.041834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.041805    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls" pod "63bff263-0915-44d7-92e1-781e193fdf12" (UID: "63bff263-0915-44d7-92e1-781e193fdf12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:08.041834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.041826    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs" pod "63bff263-0915-44d7-92e1-781e193fdf12" (UID: "63bff263-0915-44d7-92e1-781e193fdf12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:08.042044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.041908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn" pod "63bff263-0915-44d7-92e1-781e193fdf12" (UID: "63bff263-0915-44d7-92e1-781e193fdf12"). InnerVolumeSpecName "kube-api-access-k6rqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:08.141789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.141703    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:08.141789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.141736    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bff263-0915-44d7-92e1-781e193fdf12-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:08.141789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.141749    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rqn\" (UniqueName: \"kubernetes.io/projected/63bff263-0915-44d7-92e1-781e193fdf12-kube-api-access-k6rqn\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:08.322018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.321982    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"]
Nov 02 15:47:08.330563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:08.330533    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78d075-567d76b598-45klq"]
Nov 02 15:47:09.894935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:09.894896    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bff263-0915-44d7-92e1-781e193fdf12" path="/var/lib/kubelet/pods/63bff263-0915-44d7-92e1-781e193fdf12/volumes"
Nov 02 15:47:11.740214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.739003    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:47:11.749513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.749480    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"]
Nov 02 15:47:11.751623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751594    5041 cpu_manager.go:414] "containerMap: removing container" podUID="63bff263-0915-44d7-92e1-781e193fdf12" containerName="maas-api"
Nov 02 15:47:11.751766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751625    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bff263-0915-44d7-92e1-781e193fdf12" containerName="maas-api"
Nov 02 15:47:11.751766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751712    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing"
Nov 02 15:47:11.751766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751747    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing"
Nov 02 15:47:11.751897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751772    5041 cpu_manager.go:414] "containerMap: removing container" podUID="47da5dd3-8a24-41b1-a29d-cb8accb18eff" containerName="payload-pre-processing"
Nov 02 15:47:11.751897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751786    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47da5dd3-8a24-41b1-a29d-cb8accb18eff" containerName="payload-pre-processing"
Nov 02 15:47:11.751897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751824    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing"
Nov 02 15:47:11.751897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751834    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing"
Nov 02 15:47:11.751897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751861    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy"
Nov 02 15:47:11.751897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751870    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy"
Nov 02 15:47:11.752081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751938    5041 cpu_manager.go:414] "containerMap: removing container" podUID="65e47cda-7a73-44a1-9646-d75526f9c07c" containerName="payload-processing"
Nov 02 15:47:11.752081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.751947    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e47cda-7a73-44a1-9646-d75526f9c07c" containerName="payload-processing"
Nov 02 15:47:11.752081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752014    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d334f924-047f-4f60-8ca1-6d8751dd9833" containerName="maas-api"
Nov 02 15:47:11.752081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752023    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d334f924-047f-4f60-8ca1-6d8751dd9833" containerName="maas-api"
Nov 02 15:47:11.752331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752312    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="47da5dd3-8a24-41b1-a29d-cb8accb18eff" containerName="payload-pre-processing"
Nov 02 15:47:11.752331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752331    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing"
Nov 02 15:47:11.752508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752344    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerName="istio-proxy"
Nov 02 15:47:11.752508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752389    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e47cda-7a73-44a1-9646-d75526f9c07c" containerName="payload-processing"
Nov 02 15:47:11.752508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752408    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d334f924-047f-4f60-8ca1-6d8751dd9833" containerName="maas-api"
Nov 02 15:47:11.752508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.752426    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bff263-0915-44d7-92e1-781e193fdf12" containerName="maas-api"
Nov 02 15:47:11.758074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.758055    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.770701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.770193    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:47:11.771146 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.771112    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d79991-serving-cert\""
Nov 02 15:47:11.777152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.775294    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"]
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796046    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796121    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796166    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796196    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796215    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796237    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796254    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796281    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796300    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796343    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796387    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796513    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796597    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796628    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796668    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796718    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert\") pod \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\" (UID: \"9a50fcd5-8c60-4913-afad-6174f9d38f2b\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.796865    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket\") pod \"3b76898c-ef4e-45a3-bec8-88e611382ee7\" (UID: \"3b76898c-ef4e-45a3-bec8-88e611382ee7\") "
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.797275    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.797342    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.797957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.797447    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjp4\" (UniqueName: \"kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.798880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.798044    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.798880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.798292    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.798880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.798356    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.798880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.798579    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.798880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.798745    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.799117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.799012    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:11.799864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.799835    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:11.804658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.804382    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.804771 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.804675    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.804936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.804913    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805057    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805193    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805211    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805225    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805241    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805255    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805268    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805283    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805296    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805311    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.805320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.805324    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b76898c-ef4e-45a3-bec8-88e611382ee7-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.834446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.828983    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:11.834446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.829076    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.834446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.829085    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "kube-api-access-bchsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.834446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.829597    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.834446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.829882    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "kube-api-access-gskr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.834446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.830283    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:11.834966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.834713    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token" pod "9a50fcd5-8c60-4913-afad-6174f9d38f2b" (UID: "9a50fcd5-8c60-4913-afad-6174f9d38f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.840584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.838867    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy" pod "3b76898c-ef4e-45a3-bec8-88e611382ee7" (UID: "3b76898c-ef4e-45a3-bec8-88e611382ee7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.849806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.849778    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906320    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906459    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906528    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906620    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906677    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906710    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.906758    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.907121    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.908411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.907182    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs\") pod \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\" (UID: \"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0\") "
Nov 02 15:47:11.911115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.911082    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.911350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.911333    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjp4\" (UniqueName: \"kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.911807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.911617    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.912282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.912118    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.926990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.926954    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.927884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.927849    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:11.933224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.931215    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "kube-api-access-qg6zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.933914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.933869    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:11.934031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.933991    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.934208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.934178    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.934439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.934398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.934594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.934568    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.937627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.937593    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy" pod "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" (UID: "8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.960488 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:11.949344    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-serving-cert: secret "maas-api-e2e-ipp-a-d79991-serving-cert" not found
Nov 02 15:47:11.960648 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:11.960535    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls podName:a394f683-c6d7-42cb-82d2-5fe27e0ccb9a nodeName:}" failed. No retries permitted until 2026-11-02 15:47:12.460507977 +0000 UTC m=+3084.718577407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls") pod "maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" (UID: "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a") : secret "maas-api-e2e-ipp-a-d79991-serving-cert" not found
Nov 02 15:47:11.960648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.950144    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.960648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960582    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.960648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960601    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.960648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960616    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a50fcd5-8c60-4913-afad-6174f9d38f2b-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.960648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960631    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchsd\" (UniqueName: \"kubernetes.io/projected/9a50fcd5-8c60-4913-afad-6174f9d38f2b-kube-api-access-bchsd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.961088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960650    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gskr5\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-kube-api-access-gskr5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.961088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960668    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.961088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.960682    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b76898c-ef4e-45a3-bec8-88e611382ee7-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.961088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.950028    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.964819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.964758    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:11.972796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:11.972764    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjp4\" (UniqueName: \"kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:12.011319 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.011288    5041 generic.go:356] "Generic (PLEG): container finished" podID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerID="74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa" exitCode=0
Nov 02 15:47:12.011475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.011381    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"
Nov 02 15:47:12.011475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.011407    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" event={"ID":"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0","Type":"ContainerDied","Data":"74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa"}
Nov 02 15:47:12.011475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.011469    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs" event={"ID":"8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0","Type":"ContainerDied","Data":"53949e4f1dff30e8f8e2217c91fee93db11ec33803602d22f7821f791f36b5c6"}
Nov 02 15:47:12.011647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.011497    5041 scope.go:122] "RemoveContainer" containerID="74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa"
Nov 02 15:47:12.013274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.013251    5041 generic.go:356] "Generic (PLEG): container finished" podID="3b76898c-ef4e-45a3-bec8-88e611382ee7" containerID="a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7" exitCode=0
Nov 02 15:47:12.013528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.013504    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" event={"ID":"3b76898c-ef4e-45a3-bec8-88e611382ee7","Type":"ContainerDied","Data":"a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7"}
Nov 02 15:47:12.013656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.013558    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6" event={"ID":"3b76898c-ef4e-45a3-bec8-88e611382ee7","Type":"ContainerDied","Data":"ef5fa3f7b80e69e41f09b60226c7183fc9a4adaaa5f7c64cb287649c1760d92d"}
Nov 02 15:47:12.013656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.013643    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"
Nov 02 15:47:12.015972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.015944    5041 generic.go:356] "Generic (PLEG): container finished" podID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerID="a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc" exitCode=0
Nov 02 15:47:12.016098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.015997    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" event={"ID":"9a50fcd5-8c60-4913-afad-6174f9d38f2b","Type":"ContainerDied","Data":"a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc"}
Nov 02 15:47:12.016098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.016037    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd" event={"ID":"9a50fcd5-8c60-4913-afad-6174f9d38f2b","Type":"ContainerDied","Data":"b954c25f1cd04674ab4947783a36ad917ba6cb9c0a01316f3b6e31a1f554df26"}
Nov 02 15:47:12.016228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.016152    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"
Nov 02 15:47:12.032566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.032543    5041 scope.go:122] "RemoveContainer" containerID="74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa"
Nov 02 15:47:12.032883 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:12.032854    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa\": container with ID starting with 74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa not found: ID does not exist" containerID="74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa"
Nov 02 15:47:12.032972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.032893    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa"} err="failed to get container status \"74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa\": rpc error: code = NotFound desc = could not find container \"74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa\": container with ID starting with 74498bf3cd1dfdbe90b449639ff95a7e760768eff6cd6d27c6e185ef3b399faa not found: ID does not exist"
Nov 02 15:47:12.032972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.032916    5041 scope.go:122] "RemoveContainer" containerID="a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7"
Nov 02 15:47:12.043814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.043789    5041 scope.go:122] "RemoveContainer" containerID="a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7"
Nov 02 15:47:12.044101 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:12.044081    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7\": container with ID starting with a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7 not found: ID does not exist" containerID="a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7"
Nov 02 15:47:12.044166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.044107    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7"} err="failed to get container status \"a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7\": rpc error: code = NotFound desc = could not find container \"a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7\": container with ID starting with a38dbea5da059cf6fd9b0e3f00bc75d24e17b18fd26323931d8793edc1c574c7 not found: ID does not exist"
Nov 02 15:47:12.044166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.044124    5041 scope.go:122] "RemoveContainer" containerID="a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc"
Nov 02 15:47:12.055734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.055698    5041 scope.go:122] "RemoveContainer" containerID="a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc"
Nov 02 15:47:12.056298 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:12.056261    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc\": container with ID starting with a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc not found: ID does not exist" containerID="a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc"
Nov 02 15:47:12.056455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.056307    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc"} err="failed to get container status \"a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc\": rpc error: code = NotFound desc = could not find container \"a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc\": container with ID starting with a28921d3fad901d0c402bd3a17226afd48cf7340ebf58e3c93dee07e2f4f51cc not found: ID does not exist"
Nov 02 15:47:12.059585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.059562    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"]
Nov 02 15:47:12.062188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062154    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062181    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062198    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062210    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg6zs\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-kube-api-access-qg6zs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062225    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062240    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062254    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062268    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.062350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.062281    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.065273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.065240    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-q6lcs"]
Nov 02 15:47:12.094802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.094777    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"]
Nov 02 15:47:12.100729 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.100706    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-v2lnd"]
Nov 02 15:47:12.132237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.132207    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"]
Nov 02 15:47:12.138695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.138663    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-w4zn6"]
Nov 02 15:47:12.468026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.467995    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:12.470516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.470486    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:12.705129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:12.705101    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:13.015180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:13.015148    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"]
Nov 02 15:47:13.015547 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:47:13.015338    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394f683_c6d7_42cb_82d2_5fe27e0ccb9a.slice/crio-655d788395e35330016d05cdbf96624e9d55d9650e5ffab2ffe41390ad854558 WatchSource:0}: Error finding container 655d788395e35330016d05cdbf96624e9d55d9650e5ffab2ffe41390ad854558: Status 404 returned error can't find the container with id 655d788395e35330016d05cdbf96624e9d55d9650e5ffab2ffe41390ad854558
Nov 02 15:47:13.029605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:13.029475    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" event={"ID":"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a","Type":"ContainerStarted","Data":"655d788395e35330016d05cdbf96624e9d55d9650e5ffab2ffe41390ad854558"}
Nov 02 15:47:13.891840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:13.891805    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b76898c-ef4e-45a3-bec8-88e611382ee7" path="/var/lib/kubelet/pods/3b76898c-ef4e-45a3-bec8-88e611382ee7/volumes"
Nov 02 15:47:13.892373 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:13.892347    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" path="/var/lib/kubelet/pods/8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0/volumes"
Nov 02 15:47:13.892828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:13.892814    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" path="/var/lib/kubelet/pods/9a50fcd5-8c60-4913-afad-6174f9d38f2b/volumes"
Nov 02 15:47:14.040916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:14.040874    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" event={"ID":"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a","Type":"ContainerStarted","Data":"5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d"}
Nov 02 15:47:14.041295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:14.040945    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:14.082763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:14.082710    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" podStartSLOduration=2.697077321 podStartE2EDuration="3.082696685s" totalImagesPullingTime="385.619364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:14.078496411 +0000 UTC m=+3086.336565873" watchObservedRunningTime="2026-11-02 15:47:14.082696685 +0000 UTC m=+3086.340766135"
Nov 02 15:47:17.579980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:17.579941    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"]
Nov 02 15:47:20.054859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:20.054825    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:47:22.581273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:22.581231    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerName="payload-processing" containerID="cri-o://1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a" gracePeriod=25
Nov 02 15:47:22.874816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:22.874723    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"]
Nov 02 15:47:28.669930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:28.669842    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"]
Nov 02 15:47:28.992422 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:28.992293    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:33.670742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:33.670700    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" containerName="payload-pre-processing" containerID="cri-o://857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07" gracePeriod=25
Nov 02 15:47:36.116524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:36.116489    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"]
Nov 02 15:47:36.169509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:36.169475    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"]
Nov 02 15:47:36.774651 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:36.774607    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:47:37.897651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:37.897610    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main" containerID="cri-o://1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0" gracePeriod=45
Nov 02 15:47:38.179667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.179640    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:47:38.244811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.244767    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache\") pod \"49c1029f-d9e4-4932-922f-8336beee4182\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") "
Nov 02 15:47:38.244988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.244850    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir\") pod \"49c1029f-d9e4-4932-922f-8336beee4182\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") "
Nov 02 15:47:38.244988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.244904    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7\" (UniqueName: \"kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7\") pod \"49c1029f-d9e4-4932-922f-8336beee4182\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") "
Nov 02 15:47:38.244988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.244935    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home\") pod \"49c1029f-d9e4-4932-922f-8336beee4182\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") "
Nov 02 15:47:38.245165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.245022    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm\") pod \"49c1029f-d9e4-4932-922f-8336beee4182\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") "
Nov 02 15:47:38.245165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.245107    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs\" (UniqueName: \"kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs\") pod \"49c1029f-d9e4-4932-922f-8336beee4182\" (UID: \"49c1029f-d9e4-4932-922f-8336beee4182\") "
Nov 02 15:47:38.246834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.246649    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache" pod "49c1029f-d9e4-4932-922f-8336beee4182" (UID: "49c1029f-d9e4-4932-922f-8336beee4182"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.247059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.247022    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir" pod "49c1029f-d9e4-4932-922f-8336beee4182" (UID: "49c1029f-d9e4-4932-922f-8336beee4182"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.247405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.247383    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home" pod "49c1029f-d9e4-4932-922f-8336beee4182" (UID: "49c1029f-d9e4-4932-922f-8336beee4182"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.251758 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.250958    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm" pod "49c1029f-d9e4-4932-922f-8336beee4182" (UID: "49c1029f-d9e4-4932-922f-8336beee4182"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.252793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.252772    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs" pod "49c1029f-d9e4-4932-922f-8336beee4182" (UID: "49c1029f-d9e4-4932-922f-8336beee4182"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:38.254465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.254440    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7" pod "49c1029f-d9e4-4932-922f-8336beee4182" (UID: "49c1029f-d9e4-4932-922f-8336beee4182"). InnerVolumeSpecName "kube-api-access-v5wh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:38.291768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.291451    5041 generic.go:356] "Generic (PLEG): container finished" podID="49c1029f-d9e4-4932-922f-8336beee4182" containerID="1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0" exitCode=0
Nov 02 15:47:38.291768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.291546    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"
Nov 02 15:47:38.291768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.291566    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" event={"ID":"49c1029f-d9e4-4932-922f-8336beee4182","Type":"ContainerDied","Data":"1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0"}
Nov 02 15:47:38.291768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.291605    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw" event={"ID":"49c1029f-d9e4-4932-922f-8336beee4182","Type":"ContainerDied","Data":"c4539b8792491b3a81415ca63b7fbf29d4c47cddecd36b036710ec29e661133d"}
Nov 02 15:47:38.291768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.291626    5041 scope.go:122] "RemoveContainer" containerID="1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0"
Nov 02 15:47:38.312082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.312057    5041 scope.go:122] "RemoveContainer" containerID="1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0"
Nov 02 15:47:38.312417 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:38.312391    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0\": container with ID starting with 1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0 not found: ID does not exist" containerID="1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0"
Nov 02 15:47:38.314142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.312554    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0"} err="failed to get container status \"1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0\": rpc error: code = NotFound desc = could not find container \"1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0\": container with ID starting with 1b74bb57e0435519b11038cff8ef82c4f2552f1664407428344ef9a868c175a0 not found: ID does not exist"
Nov 02 15:47:38.347392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.347342    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c1029f-d9e4-4932-922f-8336beee4182-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.347605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.347580    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.347700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.347610    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.347700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.347625    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5wh7\" (UniqueName: \"kubernetes.io/projected/49c1029f-d9e4-4932-922f-8336beee4182-kube-api-access-v5wh7\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.347700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.347640    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.347700 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.347656    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c1029f-d9e4-4932-922f-8336beee4182-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.351454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.351390    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"]
Nov 02 15:47:38.358436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.358387    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-2-w4-7ecfe1-kserve-6d75b958cc-m47jw"]
Nov 02 15:47:38.620973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.620930    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"]
Nov 02 15:47:38.992010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:38.991969    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:39.912388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:39.902945    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c1029f-d9e4-4932-922f-8336beee4182" path="/var/lib/kubelet/pods/49c1029f-d9e4-4932-922f-8336beee4182/volumes"
Nov 02 15:47:41.117790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.117750    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7" podUID="ba55d13c-e299-4508-8ca2-3918533ac4af" containerName="payload-processing" containerID="cri-o://1d7ff6c03c81319781ac9eafd986fde6bff08edd106c777b27b246a87e123765" gracePeriod=25
Nov 02 15:47:41.170356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.170318    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt" podUID="d2059821-9d01-4178-a072-ce18f726ec18" containerName="payload-pre-processing" containerID="cri-o://565264c00dd87bdb56be40b59e9a19cc8c11868a908b41dfbdf099a4711a0d5b" gracePeriod=25
Nov 02 15:47:41.324184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.324140    5041 generic.go:356] "Generic (PLEG): container finished" podID="d2059821-9d01-4178-a072-ce18f726ec18" containerID="565264c00dd87bdb56be40b59e9a19cc8c11868a908b41dfbdf099a4711a0d5b" exitCode=0
Nov 02 15:47:41.324414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.324210    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt" event={"ID":"d2059821-9d01-4178-a072-ce18f726ec18","Type":"ContainerDied","Data":"565264c00dd87bdb56be40b59e9a19cc8c11868a908b41dfbdf099a4711a0d5b"}
Nov 02 15:47:41.326349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.326320    5041 generic.go:356] "Generic (PLEG): container finished" podID="ba55d13c-e299-4508-8ca2-3918533ac4af" containerID="1d7ff6c03c81319781ac9eafd986fde6bff08edd106c777b27b246a87e123765" exitCode=0
Nov 02 15:47:41.326516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.326406    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7" event={"ID":"ba55d13c-e299-4508-8ca2-3918533ac4af","Type":"ContainerDied","Data":"1d7ff6c03c81319781ac9eafd986fde6bff08edd106c777b27b246a87e123765"}
Nov 02 15:47:41.478401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.478372    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:47:41.530562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.529062    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:47:41.586757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.586711    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume\") pod \"d2059821-9d01-4178-a072-ce18f726ec18\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") "
Nov 02 15:47:41.586936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.586781    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs\" (UniqueName: \"kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs\") pod \"ba55d13c-e299-4508-8ca2-3918533ac4af\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") "
Nov 02 15:47:41.586936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.586907    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm\" (UniqueName: \"kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm\") pod \"d2059821-9d01-4178-a072-ce18f726ec18\" (UID: \"d2059821-9d01-4178-a072-ce18f726ec18\") "
Nov 02 15:47:41.587111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.587094    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume\") pod \"ba55d13c-e299-4508-8ca2-3918533ac4af\" (UID: \"ba55d13c-e299-4508-8ca2-3918533ac4af\") "
Nov 02 15:47:41.587193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.587131    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume" pod "d2059821-9d01-4178-a072-ce18f726ec18" (UID: "d2059821-9d01-4178-a072-ce18f726ec18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:41.587569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.587543    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2059821-9d01-4178-a072-ce18f726ec18-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:41.587888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.587858    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume" pod "ba55d13c-e299-4508-8ca2-3918533ac4af" (UID: "ba55d13c-e299-4508-8ca2-3918533ac4af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:41.589583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.589559    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm" pod "d2059821-9d01-4178-a072-ce18f726ec18" (UID: "d2059821-9d01-4178-a072-ce18f726ec18"). InnerVolumeSpecName "kube-api-access-vl9tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:41.589723 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.589699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs" pod "ba55d13c-e299-4508-8ca2-3918533ac4af" (UID: "ba55d13c-e299-4508-8ca2-3918533ac4af"). InnerVolumeSpecName "kube-api-access-cmjvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:41.688938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.688901    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9tm\" (UniqueName: \"kubernetes.io/projected/d2059821-9d01-4178-a072-ce18f726ec18-kube-api-access-vl9tm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:41.688938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.688936    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba55d13c-e299-4508-8ca2-3918533ac4af-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:41.689144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:41.688951    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmjvs\" (UniqueName: \"kubernetes.io/projected/ba55d13c-e299-4508-8ca2-3918533ac4af-kube-api-access-cmjvs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:42.338728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.338691    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"
Nov 02 15:47:42.339215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.338694    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt" event={"ID":"d2059821-9d01-4178-a072-ce18f726ec18","Type":"ContainerDied","Data":"7974a6d2b4c8c3366cc4721f3d4a7a864acab5002c47a2a8f3721cd1902ad5e6"}
Nov 02 15:47:42.339215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.338872    5041 scope.go:122] "RemoveContainer" containerID="565264c00dd87bdb56be40b59e9a19cc8c11868a908b41dfbdf099a4711a0d5b"
Nov 02 15:47:42.342010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.341982    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7" event={"ID":"ba55d13c-e299-4508-8ca2-3918533ac4af","Type":"ContainerDied","Data":"2449df4c033371defaf8b81b55a81a4d39f7b117a4c9a2a7ca023247bb52267e"}
Nov 02 15:47:42.342464 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.342394    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"
Nov 02 15:47:42.352886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.352837    5041 scope.go:122] "RemoveContainer" containerID="1d7ff6c03c81319781ac9eafd986fde6bff08edd106c777b27b246a87e123765"
Nov 02 15:47:42.386906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.386874    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"]
Nov 02 15:47:42.406071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.406006    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-68694fb575-wxktt"]
Nov 02 15:47:42.448870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.448823    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"]
Nov 02 15:47:42.453497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.453462    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-6694b888d7-v28n7"]
Nov 02 15:47:42.466995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.466955    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:42.890549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:42.890521    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:47:43.020097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.020022    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt\" (UniqueName: \"kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt\") pod \"09975717-b3b1-46ed-82b1-be1c2ca8403e\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") "
Nov 02 15:47:43.020097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.020071    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume\") pod \"09975717-b3b1-46ed-82b1-be1c2ca8403e\" (UID: \"09975717-b3b1-46ed-82b1-be1c2ca8403e\") "
Nov 02 15:47:43.020567 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.020531    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume" pod "09975717-b3b1-46ed-82b1-be1c2ca8403e" (UID: "09975717-b3b1-46ed-82b1-be1c2ca8403e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:43.022273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.022245    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt" pod "09975717-b3b1-46ed-82b1-be1c2ca8403e" (UID: "09975717-b3b1-46ed-82b1-be1c2ca8403e"). InnerVolumeSpecName "kube-api-access-fkkgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:43.122338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.122297    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkgt\" (UniqueName: \"kubernetes.io/projected/09975717-b3b1-46ed-82b1-be1c2ca8403e-kube-api-access-fkkgt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:43.122526 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.122349    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09975717-b3b1-46ed-82b1-be1c2ca8403e-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:43.351447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.351332    5041 generic.go:356] "Generic (PLEG): container finished" podID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerID="1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a" exitCode=0
Nov 02 15:47:43.351447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.351404    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"
Nov 02 15:47:43.351447 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.351417    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" event={"ID":"09975717-b3b1-46ed-82b1-be1c2ca8403e","Type":"ContainerDied","Data":"1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a"}
Nov 02 15:47:43.352127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.351473    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf" event={"ID":"09975717-b3b1-46ed-82b1-be1c2ca8403e","Type":"ContainerDied","Data":"cdbd7c713c53be208cee6f969086c790e57fe11268458aeba6c82eb108bb6bfe"}
Nov 02 15:47:43.352127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.351501    5041 scope.go:122] "RemoveContainer" containerID="1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a"
Nov 02 15:47:43.375390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.375328    5041 scope.go:122] "RemoveContainer" containerID="1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a"
Nov 02 15:47:43.375793 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:43.375704    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a\": container with ID starting with 1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a not found: ID does not exist" containerID="1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a"
Nov 02 15:47:43.375873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.375807    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a"} err="failed to get container status \"1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a\": rpc error: code = NotFound desc = could not find container \"1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a\": container with ID starting with 1b9f403857b98e9b9f5c963280e0a8da64d1f76bd152c98da758781fbe5eef4a not found: ID does not exist"
Nov 02 15:47:43.398290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.398262    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"]
Nov 02 15:47:43.407656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.407635    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b955c5-8b498d75-n5jgf"]
Nov 02 15:47:43.890123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.890089    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" path="/var/lib/kubelet/pods/09975717-b3b1-46ed-82b1-be1c2ca8403e/volumes"
Nov 02 15:47:43.890592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.890577    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba55d13c-e299-4508-8ca2-3918533ac4af" path="/var/lib/kubelet/pods/ba55d13c-e299-4508-8ca2-3918533ac4af/volumes"
Nov 02 15:47:43.890964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:43.890951    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2059821-9d01-4178-a072-ce18f726ec18" path="/var/lib/kubelet/pods/d2059821-9d01-4178-a072-ce18f726ec18/volumes"
Nov 02 15:47:46.024236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.024201    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"]
Nov 02 15:47:46.025032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025013    5041 cpu_manager.go:414] "containerMap: removing container" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerName="payload-processing"
Nov 02 15:47:46.025079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025035    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerName="payload-processing"
Nov 02 15:47:46.025079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025049    5041 cpu_manager.go:414] "containerMap: removing container" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main"
Nov 02 15:47:46.025079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025054    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main"
Nov 02 15:47:46.025079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025072    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ba55d13c-e299-4508-8ca2-3918533ac4af" containerName="payload-processing"
Nov 02 15:47:46.025079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025077    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba55d13c-e299-4508-8ca2-3918533ac4af" containerName="payload-processing"
Nov 02 15:47:46.025246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025112    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d2059821-9d01-4178-a072-ce18f726ec18" containerName="payload-pre-processing"
Nov 02 15:47:46.025246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025117    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2059821-9d01-4178-a072-ce18f726ec18" containerName="payload-pre-processing"
Nov 02 15:47:46.025246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025126    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy"
Nov 02 15:47:46.025246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025131    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy"
Nov 02 15:47:46.025246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025150    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy"
Nov 02 15:47:46.025246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025154    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025254    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a50fcd5-8c60-4913-afad-6174f9d38f2b" containerName="istio-proxy"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025262    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="09975717-b3b1-46ed-82b1-be1c2ca8403e" containerName="payload-processing"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025271    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c1029f-d9e4-4932-922f-8336beee4182" containerName="main"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025280    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a80097a-9d77-41e7-b551-3a71731000ff" containerName="payload-processing"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025288    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba55d13c-e299-4508-8ca2-3918533ac4af" containerName="payload-processing"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025301    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2059821-9d01-4178-a072-ce18f726ec18" containerName="payload-pre-processing"
Nov 02 15:47:46.025453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.025307    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4ca564-55bd-49d1-a5ad-4d80b1bfb0c0" containerName="istio-proxy"
Nov 02 15:47:46.030555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.030534    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.055513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.055057    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e40940\""
Nov 02 15:47:46.055513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.055116    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e40940-dockercfg-qnw7z\""
Nov 02 15:47:46.055513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.055417    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"]
Nov 02 15:47:46.127382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.124191    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"]
Nov 02 15:47:46.160535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.160490    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsm7r\" (UniqueName: \"kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r\") pod \"payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.160733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.160612    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.173982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.173949    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"]
Nov 02 15:47:46.175139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.174287    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.262327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.262293    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.262518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.262495    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsm7r\" (UniqueName: \"kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r\") pod \"payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.262593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.262578    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9nt\" (UniqueName: \"kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt\") pod \"payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.262636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.262611    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.263316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.263299    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.289967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.289914    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsm7r\" (UniqueName: \"kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r\") pod \"payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.345614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.345586    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:46.364241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.364217    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9nt\" (UniqueName: \"kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt\") pod \"payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.364337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.364306    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.365008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.364990    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.387188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.387159    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9nt\" (UniqueName: \"kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt\") pod \"payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.499379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.499339    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:46.664708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.664681    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"]
Nov 02 15:47:46.665572 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:47:46.665544    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ada712_fbad_4661_9450_50a1768130d0.slice/crio-80834054a417b65cbe285b1623e848e4d837175359dd9ec97df1ee7d184b18e0 WatchSource:0}: Error finding container 80834054a417b65cbe285b1623e848e4d837175359dd9ec97df1ee7d184b18e0: Status 404 returned error can't find the container with id 80834054a417b65cbe285b1623e848e4d837175359dd9ec97df1ee7d184b18e0
Nov 02 15:47:46.819943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:46.819916    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"]
Nov 02 15:47:46.820151 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:47:46.820133    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf49d34b_c1e9_44db_aac9_ed19c59d68ea.slice/crio-6b04b710634c559fe7fe9637682b4d5ac7afb909ec2474cfcc6174e22fdc62d6 WatchSource:0}: Error finding container 6b04b710634c559fe7fe9637682b4d5ac7afb909ec2474cfcc6174e22fdc62d6: Status 404 returned error can't find the container with id 6b04b710634c559fe7fe9637682b4d5ac7afb909ec2474cfcc6174e22fdc62d6
Nov 02 15:47:47.400753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.400713    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx" event={"ID":"32ada712-fbad-4661-9450-50a1768130d0","Type":"ContainerStarted","Data":"ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c"}
Nov 02 15:47:47.401166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.400779    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx" event={"ID":"32ada712-fbad-4661-9450-50a1768130d0","Type":"ContainerStarted","Data":"80834054a417b65cbe285b1623e848e4d837175359dd9ec97df1ee7d184b18e0"}
Nov 02 15:47:47.401166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.400812    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:47.402579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.402548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r" event={"ID":"cf49d34b-c1e9-44db-aac9-ed19c59d68ea","Type":"ContainerStarted","Data":"649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42"}
Nov 02 15:47:47.402718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.402598    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r" event={"ID":"cf49d34b-c1e9-44db-aac9-ed19c59d68ea","Type":"ContainerStarted","Data":"6b04b710634c559fe7fe9637682b4d5ac7afb909ec2474cfcc6174e22fdc62d6"}
Nov 02 15:47:47.402718 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.402647    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:47.444510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.444457    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx" podStartSLOduration=2.444443087 podStartE2EDuration="2.444443087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:47.441003835 +0000 UTC m=+3119.699073286" watchObservedRunningTime="2026-11-02 15:47:47.444443087 +0000 UTC m=+3119.702512574"
Nov 02 15:47:47.486357 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:47.486299    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r" podStartSLOduration=1.486286486 podStartE2EDuration="1.486286486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:47.482887114 +0000 UTC m=+3119.740956564" watchObservedRunningTime="2026-11-02 15:47:47.486286486 +0000 UTC m=+3119.744355936"
Nov 02 15:47:52.466006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:52.465968    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:52.933080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:52.933047    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"]
Nov 02 15:47:53.649806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:53.649759    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" podUID="8449d6ca-7eb0-424f-b047-670e48373151" containerName="main" containerID="cri-o://eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c" gracePeriod=45
Nov 02 15:47:53.798933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:53.798887    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" podUID="8449d6ca-7eb0-424f-b047-670e48373151" containerName="main" probeResult="failure" output="Get \"https://10.132.1.14:8000/ready\": dial tcp 10.132.1.14:8000: connect: connection refused"
Nov 02 15:47:53.965387 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:53.965348    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:47:54.034294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.034261    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:47:54.065623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065591    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir\") pod \"8449d6ca-7eb0-424f-b047-670e48373151\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") "
Nov 02 15:47:54.065775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065637    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache\") pod \"8449d6ca-7eb0-424f-b047-670e48373151\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") "
Nov 02 15:47:54.065775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs\" (UniqueName: \"kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs\") pod \"8449d6ca-7eb0-424f-b047-670e48373151\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") "
Nov 02 15:47:54.065775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065755    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home\") pod \"8449d6ca-7eb0-424f-b047-670e48373151\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") "
Nov 02 15:47:54.065941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065827    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg\" (UniqueName: \"kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg\") pod \"8449d6ca-7eb0-424f-b047-670e48373151\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") "
Nov 02 15:47:54.065941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065832    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir" pod "8449d6ca-7eb0-424f-b047-670e48373151" (UID: "8449d6ca-7eb0-424f-b047-670e48373151"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:54.065941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.065886    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache" pod "8449d6ca-7eb0-424f-b047-670e48373151" (UID: "8449d6ca-7eb0-424f-b047-670e48373151"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:54.066106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.066001    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm\") pod \"8449d6ca-7eb0-424f-b047-670e48373151\" (UID: \"8449d6ca-7eb0-424f-b047-670e48373151\") "
Nov 02 15:47:54.066106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.066045    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home" pod "8449d6ca-7eb0-424f-b047-670e48373151" (UID: "8449d6ca-7eb0-424f-b047-670e48373151"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:54.066570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.066550    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.066570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.066571    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.066735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.066579    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.068150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.068127    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm" pod "8449d6ca-7eb0-424f-b047-670e48373151" (UID: "8449d6ca-7eb0-424f-b047-670e48373151"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:54.068227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.068167    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg" pod "8449d6ca-7eb0-424f-b047-670e48373151" (UID: "8449d6ca-7eb0-424f-b047-670e48373151"). InnerVolumeSpecName "kube-api-access-6hbzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:54.068227 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.068192    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs" pod "8449d6ca-7eb0-424f-b047-670e48373151" (UID: "8449d6ca-7eb0-424f-b047-670e48373151"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:54.167495 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.167424    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd\" (UniqueName: \"kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd\") pod \"870134d6-03de-4a45-b54f-9576327ac3d9\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") "
Nov 02 15:47:54.167639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.167551    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume\") pod \"870134d6-03de-4a45-b54f-9576327ac3d9\" (UID: \"870134d6-03de-4a45-b54f-9576327ac3d9\") "
Nov 02 15:47:54.167893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.167870    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hbzg\" (UniqueName: \"kubernetes.io/projected/8449d6ca-7eb0-424f-b047-670e48373151-kube-api-access-6hbzg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.167893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.167881    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume" pod "870134d6-03de-4a45-b54f-9576327ac3d9" (UID: "870134d6-03de-4a45-b54f-9576327ac3d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:54.168045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.167888    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8449d6ca-7eb0-424f-b047-670e48373151-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.168045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.167924    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8449d6ca-7eb0-424f-b047-670e48373151-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.169289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.169268    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd" pod "870134d6-03de-4a45-b54f-9576327ac3d9" (UID: "870134d6-03de-4a45-b54f-9576327ac3d9"). InnerVolumeSpecName "kube-api-access-2knjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:54.268910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.268882    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870134d6-03de-4a45-b54f-9576327ac3d9-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.268910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.268906    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knjd\" (UniqueName: \"kubernetes.io/projected/870134d6-03de-4a45-b54f-9576327ac3d9-kube-api-access-2knjd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.475111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.474340    5041 generic.go:356] "Generic (PLEG): container finished" podID="870134d6-03de-4a45-b54f-9576327ac3d9" containerID="857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07" exitCode=0
Nov 02 15:47:54.475111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.474535    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" event={"ID":"870134d6-03de-4a45-b54f-9576327ac3d9","Type":"ContainerDied","Data":"857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07"}
Nov 02 15:47:54.475111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.474627    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67" event={"ID":"870134d6-03de-4a45-b54f-9576327ac3d9","Type":"ContainerDied","Data":"95516ccbb5d6d9a2982d178c98fc2e32342bd1ca40219740b43ffcf4fe90a0e1"}
Nov 02 15:47:54.475111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.474666    5041 scope.go:122] "RemoveContainer" containerID="857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07"
Nov 02 15:47:54.475111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.474830    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"
Nov 02 15:47:54.482445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.477853    5041 generic.go:356] "Generic (PLEG): container finished" podID="8449d6ca-7eb0-424f-b047-670e48373151" containerID="eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c" exitCode=0
Nov 02 15:47:54.482445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.477973    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" event={"ID":"8449d6ca-7eb0-424f-b047-670e48373151","Type":"ContainerDied","Data":"eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c"}
Nov 02 15:47:54.482445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.478020    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h" event={"ID":"8449d6ca-7eb0-424f-b047-670e48373151","Type":"ContainerDied","Data":"fcda8fe37c025e0b7e3be0f9f9c76d285251d68dbaf8f6584c156081d3b92c28"}
Nov 02 15:47:54.482445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.478123    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"
Nov 02 15:47:54.497204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.497179    5041 scope.go:122] "RemoveContainer" containerID="857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07"
Nov 02 15:47:54.497557 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:54.497535    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07\": container with ID starting with 857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07 not found: ID does not exist" containerID="857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07"
Nov 02 15:47:54.497640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.497567    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07"} err="failed to get container status \"857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07\": rpc error: code = NotFound desc = could not find container \"857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07\": container with ID starting with 857cae6f91b72de970b7bf9358fd2d96bbbe60f31195937d6691075078d17d07 not found: ID does not exist"
Nov 02 15:47:54.497640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.497585    5041 scope.go:122] "RemoveContainer" containerID="eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c"
Nov 02 15:47:54.508265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.508244    5041 scope.go:122] "RemoveContainer" containerID="eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c"
Nov 02 15:47:54.508551 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:47:54.508533    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c\": container with ID starting with eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c not found: ID does not exist" containerID="eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c"
Nov 02 15:47:54.508602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.508558    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c"} err="failed to get container status \"eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c\": rpc error: code = NotFound desc = could not find container \"eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c\": container with ID starting with eb1ff04504c327be8319f7c1bcfe7a8fee9cdb67a23c17801987cb7a014c213c not found: ID does not exist"
Nov 02 15:47:54.527269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.527238    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"]
Nov 02 15:47:54.538595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.538568    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ecfe1-64c96c74f9-x2b67"]
Nov 02 15:47:54.579899 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.579862    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"]
Nov 02 15:47:54.584523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:54.584497    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-28da58/rate-test-model-28da58-kserve-5d5f695c7c-hcx2h"]
Nov 02 15:47:55.890813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:55.890780    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8449d6ca-7eb0-424f-b047-670e48373151" path="/var/lib/kubelet/pods/8449d6ca-7eb0-424f-b047-670e48373151/volumes"
Nov 02 15:47:55.891414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:55.891390    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" path="/var/lib/kubelet/pods/870134d6-03de-4a45-b54f-9576327ac3d9/volumes"
Nov 02 15:47:57.385616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.385580    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"]
Nov 02 15:47:57.386798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.386779    5041 cpu_manager.go:414] "containerMap: removing container" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" containerName="payload-pre-processing"
Nov 02 15:47:57.386849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.386802    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" containerName="payload-pre-processing"
Nov 02 15:47:57.386894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.386873    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8449d6ca-7eb0-424f-b047-670e48373151" containerName="main"
Nov 02 15:47:57.386894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.386881    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8449d6ca-7eb0-424f-b047-670e48373151" containerName="main"
Nov 02 15:47:57.387062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.387051    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8449d6ca-7eb0-424f-b047-670e48373151" containerName="main"
Nov 02 15:47:57.387100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.387073    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="870134d6-03de-4a45-b54f-9576327ac3d9" containerName="payload-pre-processing"
Nov 02 15:47:57.393314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.393290    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.408895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.408873    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5721b1\""
Nov 02 15:47:57.413566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.413537    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"]
Nov 02 15:47:57.439277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.439251    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"]
Nov 02 15:47:57.443992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.443971    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.462848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.462826    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5721b1-dockercfg-wsjnw\""
Nov 02 15:47:57.465374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.465338    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"]
Nov 02 15:47:57.518751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.518712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcp8v\" (UniqueName: \"kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v\") pod \"payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.519239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.519214    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.621347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.621311    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcp8v\" (UniqueName: \"kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v\") pod \"payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.621514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.621456    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnvb\" (UniqueName: \"kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.621618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.621596    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.621734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.621714    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.622407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.622352    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.644423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.644353    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcp8v\" (UniqueName: \"kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v\") pod \"payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.709164 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.709140    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:57.722630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.722603    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnvb\" (UniqueName: \"kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.722730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.722684    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.723332 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.723314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.745307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.745284    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnvb\" (UniqueName: \"kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.765391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.764785    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:57.933992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:57.933955    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerName="payload-pre-processing" containerID="cri-o://5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77" gracePeriod=25
Nov 02 15:47:58.016071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.016040    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"]
Nov 02 15:47:58.016222 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:47:58.016198    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279132d9_cd42_49ef_a4e4_f8623502ecc6.slice/crio-27e83554ce4ec967cac2d895d24ace2dae9178d3d3bf75c1b931f2978071f5e1 WatchSource:0}: Error finding container 27e83554ce4ec967cac2d895d24ace2dae9178d3d3bf75c1b931f2978071f5e1: Status 404 returned error can't find the container with id 27e83554ce4ec967cac2d895d24ace2dae9178d3d3bf75c1b931f2978071f5e1
Nov 02 15:47:58.095407 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:47:58.095351    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad2e454_f47f_4e96_af2b_ffd49da0adbd.slice/crio-d8bee9ef9c311353031666b7526c7509c63188f96fc4b9c4e1018750a2bd0519 WatchSource:0}: Error finding container d8bee9ef9c311353031666b7526c7509c63188f96fc4b9c4e1018750a2bd0519: Status 404 returned error can't find the container with id d8bee9ef9c311353031666b7526c7509c63188f96fc4b9c4e1018750a2bd0519
Nov 02 15:47:58.097755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.097729    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"]
Nov 02 15:47:58.415643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.415574    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:47:58.415643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.415639    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:47:58.564697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.564657    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h" event={"ID":"5ad2e454-f47f-4e96-af2b-ffd49da0adbd","Type":"ContainerStarted","Data":"58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b"}
Nov 02 15:47:58.564882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.564757    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:47:58.564882 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.564781    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h" event={"ID":"5ad2e454-f47f-4e96-af2b-ffd49da0adbd","Type":"ContainerStarted","Data":"d8bee9ef9c311353031666b7526c7509c63188f96fc4b9c4e1018750a2bd0519"}
Nov 02 15:47:58.566596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.566561    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx" event={"ID":"279132d9-cd42-49ef-a4e4-f8623502ecc6","Type":"ContainerStarted","Data":"b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844"}
Nov 02 15:47:58.566721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.566617    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx" event={"ID":"279132d9-cd42-49ef-a4e4-f8623502ecc6","Type":"ContainerStarted","Data":"27e83554ce4ec967cac2d895d24ace2dae9178d3d3bf75c1b931f2978071f5e1"}
Nov 02 15:47:58.566721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.566643    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:47:58.604661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.604620    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h" podStartSLOduration=1.6046080539999998 podStartE2EDuration="1.604608054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:58.603627919 +0000 UTC m=+3130.861697361" watchObservedRunningTime="2026-11-02 15:47:58.604608054 +0000 UTC m=+3130.862677503"
Nov 02 15:47:58.654795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:47:58.654741    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx" podStartSLOduration=1.654725244 podStartE2EDuration="1.654725244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:58.644202445 +0000 UTC m=+3130.902271896" watchObservedRunningTime="2026-11-02 15:47:58.654725244 +0000 UTC m=+3130.912794695"
Nov 02 15:48:05.395766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:05.395714    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:48:06.770213 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:06.770175    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:48:07.093814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.093731    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"]
Nov 02 15:48:07.098847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.098824    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.127452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.127420    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-24255b-dockercfg-7x42d\""
Nov 02 15:48:07.127606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.127591    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-24255b\""
Nov 02 15:48:07.133719 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.133690    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"]
Nov 02 15:48:07.151868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.151834    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.152005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.151948    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29npz\" (UniqueName: \"kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz\") pod \"payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.253548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.253517    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.253733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.253582    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29npz\" (UniqueName: \"kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz\") pod \"payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.254263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.254236    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.279714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.279688    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29npz\" (UniqueName: \"kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz\") pod \"payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.415623 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.415552    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:07.731708 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:48:07.731676    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fcda3d_6d38_4b4a_8c64_f7724076da0a.slice/crio-852cdba1cf82637753d2b9a8792c84622cb6a95769c1b45058b4efe739038680 WatchSource:0}: Error finding container 852cdba1cf82637753d2b9a8792c84622cb6a95769c1b45058b4efe739038680: Status 404 returned error can't find the container with id 852cdba1cf82637753d2b9a8792c84622cb6a95769c1b45058b4efe739038680
Nov 02 15:48:07.731877 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:07.731852    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"]
Nov 02 15:48:08.652133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:08.652089    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p" event={"ID":"e5fcda3d-6d38-4b4a-8c64-f7724076da0a","Type":"ContainerStarted","Data":"584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608"}
Nov 02 15:48:08.652607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:08.652195    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:08.652607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:08.652221    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p" event={"ID":"e5fcda3d-6d38-4b4a-8c64-f7724076da0a","Type":"ContainerStarted","Data":"852cdba1cf82637753d2b9a8792c84622cb6a95769c1b45058b4efe739038680"}
Nov 02 15:48:08.700220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:08.700161    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p" podStartSLOduration=1.7001460339999999 podStartE2EDuration="1.700146034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:08.694707563 +0000 UTC m=+3140.952777013" watchObservedRunningTime="2026-11-02 15:48:08.700146034 +0000 UTC m=+3140.958215487"
Nov 02 15:48:09.579331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:09.579300    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:48:09.579583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:09.579455    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:48:15.394873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:15.394830    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:48:18.223398 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.223339    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:48:18.402173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.402134    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx\" (UniqueName: \"kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx\") pod \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") "
Nov 02 15:48:18.402502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.402468    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume\") pod \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\" (UID: \"1d0bf2a3-8dfb-494b-a765-3317b87c6b63\") "
Nov 02 15:48:18.402780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.402757    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume" pod "1d0bf2a3-8dfb-494b-a765-3317b87c6b63" (UID: "1d0bf2a3-8dfb-494b-a765-3317b87c6b63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:18.403045 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.403030    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:18.404354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.404329    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx" pod "1d0bf2a3-8dfb-494b-a765-3317b87c6b63" (UID: "1d0bf2a3-8dfb-494b-a765-3317b87c6b63"). InnerVolumeSpecName "kube-api-access-p45kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:18.504236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.504173    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p45kx\" (UniqueName: \"kubernetes.io/projected/1d0bf2a3-8dfb-494b-a765-3317b87c6b63-kube-api-access-p45kx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:18.755671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.755597    5041 generic.go:356] "Generic (PLEG): container finished" podID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerID="5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77" exitCode=0
Nov 02 15:48:18.755671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.755643    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"
Nov 02 15:48:18.755852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.755680    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" event={"ID":"1d0bf2a3-8dfb-494b-a765-3317b87c6b63","Type":"ContainerDied","Data":"5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77"}
Nov 02 15:48:18.755852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.755746    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g" event={"ID":"1d0bf2a3-8dfb-494b-a765-3317b87c6b63","Type":"ContainerDied","Data":"2d805122f1f9e2d7d414d8402641d88635d26339290fbe0eed15d36936f0de91"}
Nov 02 15:48:18.755852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.755777    5041 scope.go:122] "RemoveContainer" containerID="5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77"
Nov 02 15:48:18.767774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.767754    5041 scope.go:122] "RemoveContainer" containerID="5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77"
Nov 02 15:48:18.768086 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:18.768062    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77\": container with ID starting with 5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77 not found: ID does not exist" containerID="5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77"
Nov 02 15:48:18.768153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.768093    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77"} err="failed to get container status \"5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77\": rpc error: code = NotFound desc = could not find container \"5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77\": container with ID starting with 5c18ce57a0cc236a5963e3121cb3172b794f01225cf45b778a96b09e3ebb8f77 not found: ID does not exist"
Nov 02 15:48:18.812069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.812044    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"]
Nov 02 15:48:18.823167 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:18.823142    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b55da1-6f8c6c4f9-swn2g"]
Nov 02 15:48:19.662961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:19.662928    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:48:19.894168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:19.894123    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" path="/var/lib/kubelet/pods/1d0bf2a3-8dfb-494b-a765-3317b87c6b63/volumes"
Nov 02 15:48:24.725789 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.725755    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"]
Nov 02 15:48:24.726627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.726604    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerName="payload-pre-processing"
Nov 02 15:48:24.726728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.726628    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerName="payload-pre-processing"
Nov 02 15:48:24.726887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.726870    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0bf2a3-8dfb-494b-a765-3317b87c6b63" containerName="payload-pre-processing"
Nov 02 15:48:24.732509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.732490    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.750626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.750607    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e40940-serving-cert\""
Nov 02 15:48:24.757061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.757037    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"]
Nov 02 15:48:24.777134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.777099    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.777259 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.777199    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.777331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.777286    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.777448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.777428    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwms\" (UniqueName: \"kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.878073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.878040    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.878268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.878133    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwms\" (UniqueName: \"kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.878268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.878191    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.878268 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:24.878202    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-serving-cert: secret "maas-api-e2e-ipp-a-e40940-serving-cert" not found
Nov 02 15:48:24.878268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.878239    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.878579 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:24.878271    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls podName:e077bfc1-3fc9-4262-bece-e6ada8091d48 nodeName:}" failed. No retries permitted until 2026-11-02 15:48:25.378251267 +0000 UTC m=+3157.636320715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls") pod "maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" (UID: "e077bfc1-3fc9-4262-bece-e6ada8091d48") : secret "maas-api-e2e-ipp-a-e40940-serving-cert" not found
Nov 02 15:48:24.878991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.878970    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.884010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.883989    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:24.902027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:24.901999    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwms\" (UniqueName: \"kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:25.383965 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:25.383929    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:25.386478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:25.386452    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:25.647626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:25.647542    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:25.959562 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:48:25.959526    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode077bfc1_3fc9_4262_bece_e6ada8091d48.slice/crio-6d0fece76dbeab31088b6cf107d5721df01301ecbfd8cea49660f16aba676c03 WatchSource:0}: Error finding container 6d0fece76dbeab31088b6cf107d5721df01301ecbfd8cea49660f16aba676c03: Status 404 returned error can't find the container with id 6d0fece76dbeab31088b6cf107d5721df01301ecbfd8cea49660f16aba676c03
Nov 02 15:48:25.960042 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:25.960015    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"]
Nov 02 15:48:26.835177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:26.835081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" event={"ID":"e077bfc1-3fc9-4262-bece-e6ada8091d48","Type":"ContainerStarted","Data":"7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a"}
Nov 02 15:48:26.835177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:26.835156    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" event={"ID":"e077bfc1-3fc9-4262-bece-e6ada8091d48","Type":"ContainerStarted","Data":"6d0fece76dbeab31088b6cf107d5721df01301ecbfd8cea49660f16aba676c03"}
Nov 02 15:48:26.835435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:26.835193    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:26.880184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:26.880103    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" podStartSLOduration=2.405667062 podStartE2EDuration="2.880091105s" totalImagesPullingTime="474.424043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:26.879204519 +0000 UTC m=+3159.137273982" watchObservedRunningTime="2026-11-02 15:48:26.880091105 +0000 UTC m=+3159.138160556"
Nov 02 15:48:31.435467 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:31.435417    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"]
Nov 02 15:48:31.436149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:31.435656    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy" containerID="cri-o://cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2" gracePeriod=30
Nov 02 15:48:32.849325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:32.849295    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:48:36.732216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.732190    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:48:36.776751 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:36.776703    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:48:36.833286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833257    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833296    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833352    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833553    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833689    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833762    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:36.833835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833820    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.833999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.833977    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.834118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.834099    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket\") pod \"91d386b5-12f7-4590-8270-51204c1da50e\" (UID: \"91d386b5-12f7-4590-8270-51204c1da50e\") "
Nov 02 15:48:36.834631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.834604    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:36.834822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.834801    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:36.834897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.834845    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.834897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.834872    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d386b5-12f7-4590-8270-51204c1da50e-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.835008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.834933    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:36.835209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.835182    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:36.837177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.837148    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:36.837316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.837266    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:48:36.837316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.837299    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:36.838173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.838151    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps" pod "91d386b5-12f7-4590-8270-51204c1da50e" (UID: "91d386b5-12f7-4590-8270-51204c1da50e"). InnerVolumeSpecName "kube-api-access-9d6ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:36.924193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.924168    5041 generic.go:356] "Generic (PLEG): container finished" podID="91d386b5-12f7-4590-8270-51204c1da50e" containerID="cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2" exitCode=0
Nov 02 15:48:36.924320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.924211    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"
Nov 02 15:48:36.924320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.924235    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" event={"ID":"91d386b5-12f7-4590-8270-51204c1da50e","Type":"ContainerDied","Data":"cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2"}
Nov 02 15:48:36.924320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.924270    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv" event={"ID":"91d386b5-12f7-4590-8270-51204c1da50e","Type":"ContainerDied","Data":"5c87c9d7f0206aa97aa93d8f4c11addb35d9de12151c2f4a7d88af03fa7171ab"}
Nov 02 15:48:36.924320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.924291    5041 scope.go:122] "RemoveContainer" containerID="cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2"
Nov 02 15:48:36.936554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936529    5041 scope.go:122] "RemoveContainer" containerID="cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2"
Nov 02 15:48:36.936825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936802    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936828    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936845    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936859    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d386b5-12f7-4590-8270-51204c1da50e-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936873    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:36.936865    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2\": container with ID starting with cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2 not found: ID does not exist" containerID="cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2"
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936904    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2"} err="failed to get container status \"cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2\": rpc error: code = NotFound desc = could not find container \"cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2\": container with ID starting with cf9a40904b4d780cf0d854c51587bd4c7a4196452db305ad9919b5505a750fa2 not found: ID does not exist"
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936883    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d386b5-12f7-4590-8270-51204c1da50e-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.936961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.936941    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6ps\" (UniqueName: \"kubernetes.io/projected/91d386b5-12f7-4590-8270-51204c1da50e-kube-api-access-9d6ps\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:36.987273 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:36.987140    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"]
Nov 02 15:48:37.015952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.015917    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-r9wjv"]
Nov 02 15:48:37.589176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.589143    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"]
Nov 02 15:48:37.590034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.590018    5041 cpu_manager.go:414] "containerMap: removing container" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy"
Nov 02 15:48:37.590094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.590035    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy"
Nov 02 15:48:37.590183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.590173    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d386b5-12f7-4590-8270-51204c1da50e" containerName="istio-proxy"
Nov 02 15:48:37.595338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.595320    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.610083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.610050    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d79991\""
Nov 02 15:48:37.610246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.610225    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d79991-dockercfg-6z7pb\""
Nov 02 15:48:37.614932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.614898    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"]
Nov 02 15:48:37.646853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.646826    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hg9\" (UniqueName: \"kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.646989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.646895    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.747813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.747784    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.748185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.748057    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hg9\" (UniqueName: \"kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.748583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.748565    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.774943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.774921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hg9\" (UniqueName: \"kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:37.890856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.890791    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d386b5-12f7-4590-8270-51204c1da50e" path="/var/lib/kubelet/pods/91d386b5-12f7-4590-8270-51204c1da50e/volumes"
Nov 02 15:48:37.911113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:37.911084    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:38.135028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.134993    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"]
Nov 02 15:48:38.140037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.140017    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.158184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.158159    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5721b1-serving-cert\""
Nov 02 15:48:38.169459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.163440    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"]
Nov 02 15:48:38.231034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.231002    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"]
Nov 02 15:48:38.239380 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:48:38.237086    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6c57dc_9bc0_4045_b897_10c1a9f92145.slice/crio-546ddcaf124436508d2c1553df25b60b890235c6be0c8a39ea01838e5327e5a4 WatchSource:0}: Error finding container 546ddcaf124436508d2c1553df25b60b890235c6be0c8a39ea01838e5327e5a4: Status 404 returned error can't find the container with id 546ddcaf124436508d2c1553df25b60b890235c6be0c8a39ea01838e5327e5a4
Nov 02 15:48:38.255643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.255616    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46sd\" (UniqueName: \"kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.255784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.255759    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.255842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.255800    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.256003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.255981    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.357240 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.357205    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.357410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.357283    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.357472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.357432    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.357582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.357566    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46sd\" (UniqueName: \"kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.357691 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:38.357572    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-serving-cert: secret "maas-api-e2e-ipp-b-5721b1-serving-cert" not found
Nov 02 15:48:38.357691 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:38.357658    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls podName:9488db94-ed3d-4d85-af40-5c77e36c5332 nodeName:}" failed. No retries permitted until 2026-11-02 15:48:38.857619628 +0000 UTC m=+3171.115689077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls") pod "maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" (UID: "9488db94-ed3d-4d85-af40-5c77e36c5332") : secret "maas-api-e2e-ipp-b-5721b1-serving-cert" not found
Nov 02 15:48:38.358050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.358028    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.359605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.359583    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.385107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.385083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46sd\" (UniqueName: \"kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.864088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.864054    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.867605 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.867467    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:38.949439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.949401    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c" event={"ID":"5a6c57dc-9bc0-4045-b897-10c1a9f92145","Type":"ContainerStarted","Data":"b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98"}
Nov 02 15:48:38.949629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.949461    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c" event={"ID":"5a6c57dc-9bc0-4045-b897-10c1a9f92145","Type":"ContainerStarted","Data":"546ddcaf124436508d2c1553df25b60b890235c6be0c8a39ea01838e5327e5a4"}
Nov 02 15:48:38.949629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:38.949523    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:39.001678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:39.001629    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c" podStartSLOduration=2.001616185 podStartE2EDuration="2.001616185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:38.996524645 +0000 UTC m=+3171.254594095" watchObservedRunningTime="2026-11-02 15:48:39.001616185 +0000 UTC m=+3171.259685634"
Nov 02 15:48:39.056819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:39.056783    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:39.402522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:39.402491    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"]
Nov 02 15:48:39.406540 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:48:39.406499    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9488db94_ed3d_4d85_af40_5c77e36c5332.slice/crio-ce22ccf10273ece982e070d868242a288f27a7ae0a294cd29a28418c866bf4a9 WatchSource:0}: Error finding container ce22ccf10273ece982e070d868242a288f27a7ae0a294cd29a28418c866bf4a9: Status 404 returned error can't find the container with id ce22ccf10273ece982e070d868242a288f27a7ae0a294cd29a28418c866bf4a9
Nov 02 15:48:39.960314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:39.960275    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" event={"ID":"9488db94-ed3d-4d85-af40-5c77e36c5332","Type":"ContainerStarted","Data":"ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af"}
Nov 02 15:48:39.960794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:39.960339    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" event={"ID":"9488db94-ed3d-4d85-af40-5c77e36c5332","Type":"ContainerStarted","Data":"ce22ccf10273ece982e070d868242a288f27a7ae0a294cd29a28418c866bf4a9"}
Nov 02 15:48:39.960794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:39.960499    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:40.002176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:40.002125    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" podStartSLOduration=1.6503154100000001 podStartE2EDuration="2.0021117s" totalImagesPullingTime="351.79629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:40.001474681 +0000 UTC m=+3172.259544131" watchObservedRunningTime="2026-11-02 15:48:40.0021117 +0000 UTC m=+3172.260181149"
Nov 02 15:48:42.537804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.537769    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"]
Nov 02 15:48:42.542802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.542783    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.564763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.564732    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"]
Nov 02 15:48:42.570324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.570298    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8537e9\""
Nov 02 15:48:42.570324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.570298    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8537e9-dockercfg-f8f6x\""
Nov 02 15:48:42.616917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.616884    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwpb\" (UniqueName: \"kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.617091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.617001    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.718836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.718787    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.719104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.719064    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwpb\" (UniqueName: \"kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.719627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.719602    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.742596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.742564    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwpb\" (UniqueName: \"kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:42.860019 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:42.859963    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:43.174671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:43.174643    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"]
Nov 02 15:48:43.174790 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:48:43.174673    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd404e47_5b89_4bcf_875a_47e326f9ee38.slice/crio-f31a633fab018ac2044969e2ae859f0fd4629bdc72d9bb1351febc2077dca2cd WatchSource:0}: Error finding container f31a633fab018ac2044969e2ae859f0fd4629bdc72d9bb1351febc2077dca2cd: Status 404 returned error can't find the container with id f31a633fab018ac2044969e2ae859f0fd4629bdc72d9bb1351febc2077dca2cd
Nov 02 15:48:44.000960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:44.000925    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q" event={"ID":"fd404e47-5b89-4bcf-875a-47e326f9ee38","Type":"ContainerStarted","Data":"b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1"}
Nov 02 15:48:44.001424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:44.000990    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q" event={"ID":"fd404e47-5b89-4bcf-875a-47e326f9ee38","Type":"ContainerStarted","Data":"f31a633fab018ac2044969e2ae859f0fd4629bdc72d9bb1351febc2077dca2cd"}
Nov 02 15:48:44.001424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:44.001022    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:44.041296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:44.041249    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q" podStartSLOduration=2.041237453 podStartE2EDuration="2.041237453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:44.040281514 +0000 UTC m=+3176.298350959" watchObservedRunningTime="2026-11-02 15:48:44.041237453 +0000 UTC m=+3176.299306903"
Nov 02 15:48:44.531353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:44.531317    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"]
Nov 02 15:48:45.974018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:45.973988    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:48:49.963348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:49.963318    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:48:49.994152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:49.994122    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"]
Nov 02 15:48:50.034927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:50.034890    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"]
Nov 02 15:48:52.035439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:52.035407    5041 scope.go:122] "RemoveContainer" containerID="47726d884204321705627080157c716d6ea3cf03213980985850619b810f5d02"
Nov 02 15:48:54.994920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:54.994877    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" podUID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" containerName="payload-processing" containerID="cri-o://e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96" gracePeriod=25
Nov 02 15:48:55.016256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.016233    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:48:55.035685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.035650    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerName="payload-pre-processing" containerID="cri-o://c093bb1b5a7ec72e223280bef7681b869c8dafd9ede251c25811bc122ca3408b" gracePeriod=25
Nov 02 15:48:55.292571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.292543    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:48:55.384080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.384046    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume\") pod \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") "
Nov 02 15:48:55.384386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.384349    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h\" (UniqueName: \"kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h\") pod \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\" (UID: \"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2\") "
Nov 02 15:48:55.385220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.384692    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume" pod "44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" (UID: "44955817-1b51-4bbf-a3f9-3c3fbb2a16c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:55.385547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.385517    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:55.387101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.387067    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h" pod "44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" (UID: "44955817-1b51-4bbf-a3f9-3c3fbb2a16c2"). InnerVolumeSpecName "kube-api-access-wxx2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:55.487294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:55.487260    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxx2h\" (UniqueName: \"kubernetes.io/projected/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2-kube-api-access-wxx2h\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:56.114355 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.114320    5041 generic.go:356] "Generic (PLEG): container finished" podID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" containerID="e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96" exitCode=0
Nov 02 15:48:56.114802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.114381    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"
Nov 02 15:48:56.114802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.114394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" event={"ID":"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2","Type":"ContainerDied","Data":"e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96"}
Nov 02 15:48:56.114802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.114454    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv" event={"ID":"44955817-1b51-4bbf-a3f9-3c3fbb2a16c2","Type":"ContainerDied","Data":"5d733165e1319ef10439ac5d3e20159d7f127c02e16b227b3721b6fe45277a3f"}
Nov 02 15:48:56.114802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.114485    5041 scope.go:122] "RemoveContainer" containerID="e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96"
Nov 02 15:48:56.125791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.125769    5041 scope.go:122] "RemoveContainer" containerID="e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96"
Nov 02 15:48:56.126120 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:48:56.126097    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96\": container with ID starting with e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96 not found: ID does not exist" containerID="e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96"
Nov 02 15:48:56.126229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.126128    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96"} err="failed to get container status \"e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96\": rpc error: code = NotFound desc = could not find container \"e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96\": container with ID starting with e34fd04acc114236842f92548b514a4a0c8bccf254407fe0935f8d3341b65c96 not found: ID does not exist"
Nov 02 15:48:56.167088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.167034    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"]
Nov 02 15:48:56.178295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:56.178271    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5113c2-8579688bd5-swjvv"]
Nov 02 15:48:57.891336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:57.891303    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" path="/var/lib/kubelet/pods/44955817-1b51-4bbf-a3f9-3c3fbb2a16c2/volumes"
Nov 02 15:48:59.502304 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.502200    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"]
Nov 02 15:48:59.503122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.503100    5041 cpu_manager.go:414] "containerMap: removing container" podUID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" containerName="payload-processing"
Nov 02 15:48:59.503122 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.503119    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" containerName="payload-processing"
Nov 02 15:48:59.503291 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.503245    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="44955817-1b51-4bbf-a3f9-3c3fbb2a16c2" containerName="payload-processing"
Nov 02 15:48:59.508835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.508792    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.528752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.528727    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"]
Nov 02 15:48:59.534593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.534571    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-af33a6\""
Nov 02 15:48:59.534842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.534823    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-af33a6-dockercfg-6f8pj\""
Nov 02 15:48:59.539711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.539686    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.539807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.539775    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwgw\" (UniqueName: \"kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.554120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.554084    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main" containerID="cri-o://efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0" gracePeriod=45
Nov 02 15:48:59.643081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.642994    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.643278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.643082    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwgw\" (UniqueName: \"kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.644487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.644455    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.669376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.669334    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwgw\" (UniqueName: \"kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.830244 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.830211    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:48:59.878516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.878460    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:48:59.946914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.946878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir\") pod \"3fbee385-a651-4bc8-82cd-c35a08875293\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") "
Nov 02 15:48:59.947103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.946928    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache\") pod \"3fbee385-a651-4bc8-82cd-c35a08875293\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") "
Nov 02 15:48:59.947103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.946993    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs\") pod \"3fbee385-a651-4bc8-82cd-c35a08875293\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") "
Nov 02 15:48:59.947311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.947140    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home\") pod \"3fbee385-a651-4bc8-82cd-c35a08875293\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") "
Nov 02 15:48:59.947311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.947192    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z\" (UniqueName: \"kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z\") pod \"3fbee385-a651-4bc8-82cd-c35a08875293\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") "
Nov 02 15:48:59.947311 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.947239    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm\") pod \"3fbee385-a651-4bc8-82cd-c35a08875293\" (UID: \"3fbee385-a651-4bc8-82cd-c35a08875293\") "
Nov 02 15:48:59.947571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.947459    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache" pod "3fbee385-a651-4bc8-82cd-c35a08875293" (UID: "3fbee385-a651-4bc8-82cd-c35a08875293"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:59.948165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.947894    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home" pod "3fbee385-a651-4bc8-82cd-c35a08875293" (UID: "3fbee385-a651-4bc8-82cd-c35a08875293"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:59.948165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.948042    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.948165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.948061    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.948165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.948114    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir" pod "3fbee385-a651-4bc8-82cd-c35a08875293" (UID: "3fbee385-a651-4bc8-82cd-c35a08875293"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:59.950599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.950492    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm" pod "3fbee385-a651-4bc8-82cd-c35a08875293" (UID: "3fbee385-a651-4bc8-82cd-c35a08875293"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:59.950599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.950557    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z" pod "3fbee385-a651-4bc8-82cd-c35a08875293" (UID: "3fbee385-a651-4bc8-82cd-c35a08875293"). InnerVolumeSpecName "kube-api-access-f982z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:59.958347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:48:59.958314    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs" pod "3fbee385-a651-4bc8-82cd-c35a08875293" (UID: "3fbee385-a651-4bc8-82cd-c35a08875293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:00.050127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.050033    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f982z\" (UniqueName: \"kubernetes.io/projected/3fbee385-a651-4bc8-82cd-c35a08875293-kube-api-access-f982z\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.050127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.050064    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.050127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.050077    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbee385-a651-4bc8-82cd-c35a08875293-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.050127 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.050091    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbee385-a651-4bc8-82cd-c35a08875293-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.141596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.141562    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"]
Nov 02 15:49:00.141775 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:49:00.141602    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7234bcbd_9e17_46b9_9407_207d8102ff30.slice/crio-ad4b5cb424e6f1ca087c13c4950bf69a666ea9a8f54eac6768f4830c4252eb43 WatchSource:0}: Error finding container ad4b5cb424e6f1ca087c13c4950bf69a666ea9a8f54eac6768f4830c4252eb43: Status 404 returned error can't find the container with id ad4b5cb424e6f1ca087c13c4950bf69a666ea9a8f54eac6768f4830c4252eb43
Nov 02 15:49:00.151971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.151900    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl" event={"ID":"7234bcbd-9e17-46b9-9407-207d8102ff30","Type":"ContainerStarted","Data":"ad4b5cb424e6f1ca087c13c4950bf69a666ea9a8f54eac6768f4830c4252eb43"}
Nov 02 15:49:00.153343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.153315    5041 generic.go:356] "Generic (PLEG): container finished" podID="3fbee385-a651-4bc8-82cd-c35a08875293" containerID="efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0" exitCode=0
Nov 02 15:49:00.153458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.153385    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"
Nov 02 15:49:00.153458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.153400    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" event={"ID":"3fbee385-a651-4bc8-82cd-c35a08875293","Type":"ContainerDied","Data":"efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0"}
Nov 02 15:49:00.153595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.153467    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc" event={"ID":"3fbee385-a651-4bc8-82cd-c35a08875293","Type":"ContainerDied","Data":"56eb2917a816dd9bd8ae3eef380f3d02d4bafa4aa1506917286196017ebbe417"}
Nov 02 15:49:00.153595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.153493    5041 scope.go:122] "RemoveContainer" containerID="efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0"
Nov 02 15:49:00.170219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.170194    5041 scope.go:122] "RemoveContainer" containerID="efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0"
Nov 02 15:49:00.170868 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:00.170727    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0\": container with ID starting with efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0 not found: ID does not exist" containerID="efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0"
Nov 02 15:49:00.170868 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.170771    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0"} err="failed to get container status \"efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0\": rpc error: code = NotFound desc = could not find container \"efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0\": container with ID starting with efc118d1345384bcf9bb504669b11339d40d98e6104afedba187078898764fd0 not found: ID does not exist"
Nov 02 15:49:00.222228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.222198    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"]
Nov 02 15:49:00.230871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:00.230840    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/simulator-w6-5113c2-kserve-65b959864-qspcc"]
Nov 02 15:49:01.163713 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:01.163673    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl" event={"ID":"7234bcbd-9e17-46b9-9407-207d8102ff30","Type":"ContainerStarted","Data":"de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d"}
Nov 02 15:49:01.164187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:01.163742    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:49:01.205166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:01.205115    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl" podStartSLOduration=2.20510328 podStartE2EDuration="2.20510328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:01.203664982 +0000 UTC m=+3193.461734431" watchObservedRunningTime="2026-11-02 15:49:01.20510328 +0000 UTC m=+3193.463172730"
Nov 02 15:49:01.459587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:01.459550    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:49:01.891212 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:01.891133    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" path="/var/lib/kubelet/pods/3fbee385-a651-4bc8-82cd-c35a08875293/volumes"
Nov 02 15:49:06.770352 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:06.770316    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:49:09.962442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:09.962408    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"]
Nov 02 15:49:09.963023 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:09.962649    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c" podUID="59e0e740-c8aa-4ff0-baac-487596900702" containerName="maas-api" containerID="cri-o://ca8d1ed77aa4603c9d5022e348a2925a6a36e8a3380184f48604aa3b135fa62e" gracePeriod=30
Nov 02 15:49:10.025709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.025678    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"]
Nov 02 15:49:10.026853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.026826    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main"
Nov 02 15:49:10.026853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.026851    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main"
Nov 02 15:49:10.027082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.027066    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbee385-a651-4bc8-82cd-c35a08875293" containerName="main"
Nov 02 15:49:10.031626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.031604    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.059145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.058658    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"]
Nov 02 15:49:10.133854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.133810    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e40940-openshift-default-dockercfg-48t4g\""
Nov 02 15:49:10.174189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.173313    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"]
Nov 02 15:49:10.174189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.173580    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc" podUID="37222fad-eaa6-4a52-a010-37f4c2c41d96" containerName="payload-processing" containerID="cri-o://b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26" gracePeriod=30
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.176930    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177018    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177043    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177128    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177221    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177287    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177329    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177404    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tlpj\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.177651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.177484    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.199525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.199494    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"]
Nov 02 15:49:10.199833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.199778    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v" podUID="99dd7387-31f6-4a7f-892c-1b188b98eb4f" containerName="payload-pre-processing" containerID="cri-o://df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db" gracePeriod=30
Nov 02 15:49:10.250642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.250517    5041 generic.go:356] "Generic (PLEG): container finished" podID="59e0e740-c8aa-4ff0-baac-487596900702" containerID="ca8d1ed77aa4603c9d5022e348a2925a6a36e8a3380184f48604aa3b135fa62e" exitCode=0
Nov 02 15:49:10.250970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.250943    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c" event={"ID":"59e0e740-c8aa-4ff0-baac-487596900702","Type":"ContainerDied","Data":"ca8d1ed77aa4603c9d5022e348a2925a6a36e8a3380184f48604aa3b135fa62e"}
Nov 02 15:49:10.279724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.279694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.279867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.279830    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.279951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.279923    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280235    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280291    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280176    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280404    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280428    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tlpj\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280556    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280619    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.280853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280642    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.281085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.280915    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.281260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.281242    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.283073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.283046    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.283517 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.283491    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.303835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.303811    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.303932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.303886    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tlpj\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.355652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.355619    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"]
Nov 02 15:49:10.357928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.357897    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:10.375324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.374633    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:49:10.396953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.396894    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"]
Nov 02 15:49:10.501917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.501845    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls\") pod \"59e0e740-c8aa-4ff0-baac-487596900702\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") "
Nov 02 15:49:10.502058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.501953    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs\") pod \"59e0e740-c8aa-4ff0-baac-487596900702\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") "
Nov 02 15:49:10.502253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.502228    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca\") pod \"59e0e740-c8aa-4ff0-baac-487596900702\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") "
Nov 02 15:49:10.502326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.502290    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4\" (UniqueName: \"kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4\") pod \"59e0e740-c8aa-4ff0-baac-487596900702\" (UID: \"59e0e740-c8aa-4ff0-baac-487596900702\") "
Nov 02 15:49:10.502928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.502895    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca" pod "59e0e740-c8aa-4ff0-baac-487596900702" (UID: "59e0e740-c8aa-4ff0-baac-487596900702"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:10.510798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.510734    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs" pod "59e0e740-c8aa-4ff0-baac-487596900702" (UID: "59e0e740-c8aa-4ff0-baac-487596900702"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:10.510798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.510739    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4" pod "59e0e740-c8aa-4ff0-baac-487596900702" (UID: "59e0e740-c8aa-4ff0-baac-487596900702"). InnerVolumeSpecName "kube-api-access-clmz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:10.527864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.527833    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls" pod "59e0e740-c8aa-4ff0-baac-487596900702" (UID: "59e0e740-c8aa-4ff0-baac-487596900702"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:10.541417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.534076    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:49:10.594670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.594641    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:49:10.604152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.604122    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e0e740-c8aa-4ff0-baac-487596900702-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.604152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.604149    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clmz4\" (UniqueName: \"kubernetes.io/projected/59e0e740-c8aa-4ff0-baac-487596900702-kube-api-access-clmz4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.604380 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.604161    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.604380 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.604169    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59e0e740-c8aa-4ff0-baac-487596900702-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.705602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.705568    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume\") pod \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") "
Nov 02 15:49:10.705794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.705653    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v\" (UniqueName: \"kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v\") pod \"37222fad-eaa6-4a52-a010-37f4c2c41d96\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") "
Nov 02 15:49:10.705794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.705748    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt\" (UniqueName: \"kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt\") pod \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\" (UID: \"99dd7387-31f6-4a7f-892c-1b188b98eb4f\") "
Nov 02 15:49:10.705794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.705778    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume\") pod \"37222fad-eaa6-4a52-a010-37f4c2c41d96\" (UID: \"37222fad-eaa6-4a52-a010-37f4c2c41d96\") "
Nov 02 15:49:10.706912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.706524    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume" pod "37222fad-eaa6-4a52-a010-37f4c2c41d96" (UID: "37222fad-eaa6-4a52-a010-37f4c2c41d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:10.708302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.708267    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume" pod "99dd7387-31f6-4a7f-892c-1b188b98eb4f" (UID: "99dd7387-31f6-4a7f-892c-1b188b98eb4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:10.709103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.709074    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v" pod "37222fad-eaa6-4a52-a010-37f4c2c41d96" (UID: "37222fad-eaa6-4a52-a010-37f4c2c41d96"). InnerVolumeSpecName "kube-api-access-q268v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:10.717698 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.717666    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt" pod "99dd7387-31f6-4a7f-892c-1b188b98eb4f" (UID: "99dd7387-31f6-4a7f-892c-1b188b98eb4f"). InnerVolumeSpecName "kube-api-access-dbjpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:10.750315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.750288    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"]
Nov 02 15:49:10.751040 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:49:10.751019    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7054c1_6dbe_4aab_8bf9_291871fba5e0.slice/crio-0e41379b744297c4d36b03a2334e98db442597891403bd5c51f929dd268fd56f WatchSource:0}: Error finding container 0e41379b744297c4d36b03a2334e98db442597891403bd5c51f929dd268fd56f: Status 404 returned error can't find the container with id 0e41379b744297c4d36b03a2334e98db442597891403bd5c51f929dd268fd56f
Nov 02 15:49:10.754608 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.754570    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:10.754701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.754658    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:10.754785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.754704    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:10.808494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.808463    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37222fad-eaa6-4a52-a010-37f4c2c41d96-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.808640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.808498    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd7387-31f6-4a7f-892c-1b188b98eb4f-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.808640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.808516    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q268v\" (UniqueName: \"kubernetes.io/projected/37222fad-eaa6-4a52-a010-37f4c2c41d96-kube-api-access-q268v\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:10.808640 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:10.808531    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjpt\" (UniqueName: \"kubernetes.io/projected/99dd7387-31f6-4a7f-892c-1b188b98eb4f-kube-api-access-dbjpt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:11.260621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.260580    5041 generic.go:356] "Generic (PLEG): container finished" podID="37222fad-eaa6-4a52-a010-37f4c2c41d96" containerID="b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26" exitCode=0
Nov 02 15:49:11.260621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.260623    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"
Nov 02 15:49:11.261081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.260750    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc" event={"ID":"37222fad-eaa6-4a52-a010-37f4c2c41d96","Type":"ContainerDied","Data":"b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26"}
Nov 02 15:49:11.261081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.260801    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc" event={"ID":"37222fad-eaa6-4a52-a010-37f4c2c41d96","Type":"ContainerDied","Data":"41a4e24295845061081d21d313b47fd3f58d7cd85021f9340f9fadca1cd3bc4b"}
Nov 02 15:49:11.261081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.260831    5041 scope.go:122] "RemoveContainer" containerID="b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26"
Nov 02 15:49:11.262693 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.262673    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"
Nov 02 15:49:11.262844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.262695    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c" event={"ID":"59e0e740-c8aa-4ff0-baac-487596900702","Type":"ContainerDied","Data":"8afd9dd2a103fd21894b57169b31fcefaffde9bca56ec2f68a014699acb54a97"}
Nov 02 15:49:11.264584 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.264557    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" event={"ID":"9d7054c1-6dbe-4aab-8bf9-291871fba5e0","Type":"ContainerStarted","Data":"59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711"}
Nov 02 15:49:11.264893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.264607    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" event={"ID":"9d7054c1-6dbe-4aab-8bf9-291871fba5e0","Type":"ContainerStarted","Data":"0e41379b744297c4d36b03a2334e98db442597891403bd5c51f929dd268fd56f"}
Nov 02 15:49:11.270379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.270339    5041 generic.go:356] "Generic (PLEG): container finished" podID="99dd7387-31f6-4a7f-892c-1b188b98eb4f" containerID="df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db" exitCode=0
Nov 02 15:49:11.270499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.270404    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v" event={"ID":"99dd7387-31f6-4a7f-892c-1b188b98eb4f","Type":"ContainerDied","Data":"df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db"}
Nov 02 15:49:11.270499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.270461    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v" event={"ID":"99dd7387-31f6-4a7f-892c-1b188b98eb4f","Type":"ContainerDied","Data":"d5fa6b1612e2a990fde0d59644758eb8ef74f065cbbb6cba5a6a19680cdc1223"}
Nov 02 15:49:11.270499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.270391    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"
Nov 02 15:49:11.275134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.274964    5041 scope.go:122] "RemoveContainer" containerID="b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26"
Nov 02 15:49:11.275294 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:11.275251    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26\": container with ID starting with b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26 not found: ID does not exist" containerID="b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26"
Nov 02 15:49:11.275337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.275289    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26"} err="failed to get container status \"b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26\": rpc error: code = NotFound desc = could not find container \"b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26\": container with ID starting with b72c2ddc9e23afaeda5d64efc6c49f6f06e6812b12ec4bb634d78fbf3d1f8a26 not found: ID does not exist"
Nov 02 15:49:11.275337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.275313    5041 scope.go:122] "RemoveContainer" containerID="ca8d1ed77aa4603c9d5022e348a2925a6a36e8a3380184f48604aa3b135fa62e"
Nov 02 15:49:11.287063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.287044    5041 scope.go:122] "RemoveContainer" containerID="df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db"
Nov 02 15:49:11.297380 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.297344    5041 scope.go:122] "RemoveContainer" containerID="df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db"
Nov 02 15:49:11.297647 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:11.297628    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db\": container with ID starting with df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db not found: ID does not exist" containerID="df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db"
Nov 02 15:49:11.297703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.297654    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db"} err="failed to get container status \"df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db\": rpc error: code = NotFound desc = could not find container \"df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db\": container with ID starting with df7c212f1f561a7cdeceadf38ca0436ee7fc46c7839df52e121743fb3d5cb2db not found: ID does not exist"
Nov 02 15:49:11.315213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.315183    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"]
Nov 02 15:49:11.320604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.320582    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dd139-78db4b47d6-x5h7c"]
Nov 02 15:49:11.358327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.358305    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:11.360250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.360183    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podStartSLOduration=2.360168851 podStartE2EDuration="2.360168851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:11.357659554 +0000 UTC m=+3203.615729003" watchObservedRunningTime="2026-11-02 15:49:11.360168851 +0000 UTC m=+3203.618238307"
Nov 02 15:49:11.360414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.360391    5041 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Nov 02 15:49:11.360475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.360443    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Nov 02 15:49:11.418743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.418710    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"]
Nov 02 15:49:11.425229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.425198    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-5796547794-r46xc"]
Nov 02 15:49:11.456995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.456965    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"]
Nov 02 15:49:11.459208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.459170    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:49:11.463159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.463133    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dd139-7848d9b6b7-nqw2v"]
Nov 02 15:49:11.890337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.890303    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37222fad-eaa6-4a52-a010-37f4c2c41d96" path="/var/lib/kubelet/pods/37222fad-eaa6-4a52-a010-37f4c2c41d96/volumes"
Nov 02 15:49:11.890875 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.890856    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e0e740-c8aa-4ff0-baac-487596900702" path="/var/lib/kubelet/pods/59e0e740-c8aa-4ff0-baac-487596900702/volumes"
Nov 02 15:49:11.891247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:11.891233    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99dd7387-31f6-4a7f-892c-1b188b98eb4f" path="/var/lib/kubelet/pods/99dd7387-31f6-4a7f-892c-1b188b98eb4f/volumes"
Nov 02 15:49:12.176130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:12.176102    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:49:12.358475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:12.358441    5041 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Nov 02 15:49:12.358887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:12.358499    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Nov 02 15:49:13.358430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:13.358394    5041 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Nov 02 15:49:13.358609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:13.358445    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Nov 02 15:49:14.358795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:14.358757    5041 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Nov 02 15:49:14.359194 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:14.358816    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Nov 02 15:49:15.349179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.349146    5041 generic.go:356] "Generic (PLEG): container finished" podID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerID="c093bb1b5a7ec72e223280bef7681b869c8dafd9ede251c25811bc122ca3408b" exitCode=0
Nov 02 15:49:15.349344 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.349220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" event={"ID":"dc7446d1-5270-4146-af20-4dedf8f877a8","Type":"ContainerDied","Data":"c093bb1b5a7ec72e223280bef7681b869c8dafd9ede251c25811bc122ca3408b"}
Nov 02 15:49:15.356688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.356659    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r" podUID="cf49d34b-c1e9-44db-aac9-ed19c59d68ea" containerName="payload-processing" containerID="cri-o://649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42" gracePeriod=25
Nov 02 15:49:15.358523 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.358500    5041 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Nov 02 15:49:15.358631 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.358546    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Nov 02 15:49:15.362323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.362307    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:49:15.376662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.376642    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume\") pod \"dc7446d1-5270-4146-af20-4dedf8f877a8\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") "
Nov 02 15:49:15.376780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.376765    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch\" (UniqueName: \"kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch\") pod \"dc7446d1-5270-4146-af20-4dedf8f877a8\" (UID: \"dc7446d1-5270-4146-af20-4dedf8f877a8\") "
Nov 02 15:49:15.376984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.376963    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume" pod "dc7446d1-5270-4146-af20-4dedf8f877a8" (UID: "dc7446d1-5270-4146-af20-4dedf8f877a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:15.377178 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.377159    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc7446d1-5270-4146-af20-4dedf8f877a8-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:15.378708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.378686    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch" pod "dc7446d1-5270-4146-af20-4dedf8f877a8" (UID: "dc7446d1-5270-4146-af20-4dedf8f877a8"). InnerVolumeSpecName "kube-api-access-b62ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:15.397930 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.397887    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx" podUID="32ada712-fbad-4661-9450-50a1768130d0" containerName="payload-pre-processing" containerID="cri-o://ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c" gracePeriod=25
Nov 02 15:49:15.478715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.478687    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b62ch\" (UniqueName: \"kubernetes.io/projected/dc7446d1-5270-4146-af20-4dedf8f877a8-kube-api-access-b62ch\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:15.719570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.719539    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:49:15.782247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.782214    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume\") pod \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") "
Nov 02 15:49:15.782444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.782266    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt\" (UniqueName: \"kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt\") pod \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\" (UID: \"cf49d34b-c1e9-44db-aac9-ed19c59d68ea\") "
Nov 02 15:49:15.783414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.783353    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume" pod "cf49d34b-c1e9-44db-aac9-ed19c59d68ea" (UID: "cf49d34b-c1e9-44db-aac9-ed19c59d68ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:15.785120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.785091    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt" pod "cf49d34b-c1e9-44db-aac9-ed19c59d68ea" (UID: "cf49d34b-c1e9-44db-aac9-ed19c59d68ea"). InnerVolumeSpecName "kube-api-access-dx9nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:15.807529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.807496    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:49:15.883755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.883673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume\") pod \"32ada712-fbad-4661-9450-50a1768130d0\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") "
Nov 02 15:49:15.883916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.883755    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r\" (UniqueName: \"kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r\") pod \"32ada712-fbad-4661-9450-50a1768130d0\" (UID: \"32ada712-fbad-4661-9450-50a1768130d0\") "
Nov 02 15:49:15.884125 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.884101    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume" pod "32ada712-fbad-4661-9450-50a1768130d0" (UID: "32ada712-fbad-4661-9450-50a1768130d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:15.884481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.884457    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:15.884575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.884484    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx9nt\" (UniqueName: \"kubernetes.io/projected/cf49d34b-c1e9-44db-aac9-ed19c59d68ea-kube-api-access-dx9nt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:15.884575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.884498    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ada712-fbad-4661-9450-50a1768130d0-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:15.886089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.886059    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r" pod "32ada712-fbad-4661-9450-50a1768130d0" (UID: "32ada712-fbad-4661-9450-50a1768130d0"). InnerVolumeSpecName "kube-api-access-wsm7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:15.986275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:15.986234    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsm7r\" (UniqueName: \"kubernetes.io/projected/32ada712-fbad-4661-9450-50a1768130d0-kube-api-access-wsm7r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:16.358639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358599    5041 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Nov 02 15:49:16.358817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358663    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Nov 02 15:49:16.358817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358718    5041 generic.go:356] "Generic (PLEG): container finished" podID="32ada712-fbad-4661-9450-50a1768130d0" containerID="ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c" exitCode=0
Nov 02 15:49:16.358817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358784    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"
Nov 02 15:49:16.358959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx" event={"ID":"32ada712-fbad-4661-9450-50a1768130d0","Type":"ContainerDied","Data":"ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c"}
Nov 02 15:49:16.358959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358877    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx" event={"ID":"32ada712-fbad-4661-9450-50a1768130d0","Type":"ContainerDied","Data":"80834054a417b65cbe285b1623e848e4d837175359dd9ec97df1ee7d184b18e0"}
Nov 02 15:49:16.358959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.358909    5041 scope.go:122] "RemoveContainer" containerID="ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c"
Nov 02 15:49:16.360732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.360714    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"
Nov 02 15:49:16.360977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.360746    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq" event={"ID":"dc7446d1-5270-4146-af20-4dedf8f877a8","Type":"ContainerDied","Data":"1d29435784b3a8b5d697982e90cbcb130820748ddd3111cac6343701ffd096e9"}
Nov 02 15:49:16.362703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.362679    5041 generic.go:356] "Generic (PLEG): container finished" podID="cf49d34b-c1e9-44db-aac9-ed19c59d68ea" containerID="649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42" exitCode=0
Nov 02 15:49:16.363080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.362734    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r" event={"ID":"cf49d34b-c1e9-44db-aac9-ed19c59d68ea","Type":"ContainerDied","Data":"649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42"}
Nov 02 15:49:16.363080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.362784    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"
Nov 02 15:49:16.363080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.362794    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r" event={"ID":"cf49d34b-c1e9-44db-aac9-ed19c59d68ea","Type":"ContainerDied","Data":"6b04b710634c559fe7fe9637682b4d5ac7afb909ec2474cfcc6174e22fdc62d6"}
Nov 02 15:49:16.371799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.371780    5041 scope.go:122] "RemoveContainer" containerID="ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c"
Nov 02 15:49:16.372056 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:16.372036    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c\": container with ID starting with ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c not found: ID does not exist" containerID="ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c"
Nov 02 15:49:16.372111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.372067    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c"} err="failed to get container status \"ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c\": rpc error: code = NotFound desc = could not find container \"ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c\": container with ID starting with ef23361f02d4f3ffdd796aebeb29d099700415013a6b761a4e898d15f0f0007c not found: ID does not exist"
Nov 02 15:49:16.372111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.372085    5041 scope.go:122] "RemoveContainer" containerID="c093bb1b5a7ec72e223280bef7681b869c8dafd9ede251c25811bc122ca3408b"
Nov 02 15:49:16.381872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.381852    5041 scope.go:122] "RemoveContainer" containerID="649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42"
Nov 02 15:49:16.392458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.392432    5041 scope.go:122] "RemoveContainer" containerID="649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42"
Nov 02 15:49:16.392720 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:16.392697    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42\": container with ID starting with 649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42 not found: ID does not exist" containerID="649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42"
Nov 02 15:49:16.392808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.392730    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42"} err="failed to get container status \"649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42\": rpc error: code = NotFound desc = could not find container \"649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42\": container with ID starting with 649473c769498774c74d8b5056c57b42cfa263760f0ef5b7f9c4a354047def42 not found: ID does not exist"
Nov 02 15:49:16.433386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.414200    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"]
Nov 02 15:49:16.441430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.441403    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-756dd58d9c-hxz2r"]
Nov 02 15:49:16.459285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.459259    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"]
Nov 02 15:49:16.465650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.465627    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-6cc5c4f499-2dtsx"]
Nov 02 15:49:16.498754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.498734    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"]
Nov 02 15:49:16.504545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:16.504518    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5113c2-659b66c94c-mgrlq"]
Nov 02 15:49:17.363995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:17.363956    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:17.364645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:17.364479    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:17.365284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:17.365257    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:49:17.891553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:17.891517    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ada712-fbad-4661-9450-50a1768130d0" path="/var/lib/kubelet/pods/32ada712-fbad-4661-9450-50a1768130d0/volumes"
Nov 02 15:49:17.892184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:17.892162    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf49d34b-c1e9-44db-aac9-ed19c59d68ea" path="/var/lib/kubelet/pods/cf49d34b-c1e9-44db-aac9-ed19c59d68ea/volumes"
Nov 02 15:49:17.892735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:17.892718    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" path="/var/lib/kubelet/pods/dc7446d1-5270-4146-af20-4dedf8f877a8/volumes"
Nov 02 15:49:19.904460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.904428    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"]
Nov 02 15:49:19.905324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905298    5041 cpu_manager.go:414] "containerMap: removing container" podUID="37222fad-eaa6-4a52-a010-37f4c2c41d96" containerName="payload-processing"
Nov 02 15:49:19.905324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905323    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37222fad-eaa6-4a52-a010-37f4c2c41d96" containerName="payload-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905350    5041 cpu_manager.go:414] "containerMap: removing container" podUID="59e0e740-c8aa-4ff0-baac-487596900702" containerName="maas-api"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905356    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e0e740-c8aa-4ff0-baac-487596900702" containerName="maas-api"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905382    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cf49d34b-c1e9-44db-aac9-ed19c59d68ea" containerName="payload-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905388    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf49d34b-c1e9-44db-aac9-ed19c59d68ea" containerName="payload-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905425    5041 cpu_manager.go:414] "containerMap: removing container" podUID="99dd7387-31f6-4a7f-892c-1b188b98eb4f" containerName="payload-pre-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905430    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99dd7387-31f6-4a7f-892c-1b188b98eb4f" containerName="payload-pre-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905437    5041 cpu_manager.go:414] "containerMap: removing container" podUID="32ada712-fbad-4661-9450-50a1768130d0" containerName="payload-pre-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905442    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ada712-fbad-4661-9450-50a1768130d0" containerName="payload-pre-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905456    5041 cpu_manager.go:414] "containerMap: removing container" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerName="payload-pre-processing"
Nov 02 15:49:19.905536 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905461    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerName="payload-pre-processing"
Nov 02 15:49:19.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905638    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="37222fad-eaa6-4a52-a010-37f4c2c41d96" containerName="payload-processing"
Nov 02 15:49:19.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905648    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e0e740-c8aa-4ff0-baac-487596900702" containerName="maas-api"
Nov 02 15:49:19.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905656    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf49d34b-c1e9-44db-aac9-ed19c59d68ea" containerName="payload-processing"
Nov 02 15:49:19.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905666    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="99dd7387-31f6-4a7f-892c-1b188b98eb4f" containerName="payload-pre-processing"
Nov 02 15:49:19.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905673    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ada712-fbad-4661-9450-50a1768130d0" containerName="payload-pre-processing"
Nov 02 15:49:19.905917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.905679    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7446d1-5270-4146-af20-4dedf8f877a8" containerName="payload-pre-processing"
Nov 02 15:49:19.909622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.909600    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:19.926888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.926866    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8537e9-serving-cert\""
Nov 02 15:49:19.946071 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.946039    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgwj\" (UniqueName: \"kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:19.946307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.946286    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:19.946554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.946535    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:19.946975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.946955    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:19.947226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:19.947208    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"]
Nov 02 15:49:20.048163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.048127    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.048337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.048191    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgwj\" (UniqueName: \"kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.048337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.048261    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.048903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.048688    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.049676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.049648    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.051276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.051250    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.051417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.051327    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.071049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.071025    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgwj\" (UniqueName: \"kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj\") pod \"maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.225088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.225059    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:20.532536 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:49:20.532502    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d6fd17_fdd7_442d_a004_96d2c4e73a3b.slice/crio-271aa8660ba901f534eef6334082d3956cbd4a961013d9c57e48f3b78a4b5573 WatchSource:0}: Error finding container 271aa8660ba901f534eef6334082d3956cbd4a961013d9c57e48f3b78a4b5573: Status 404 returned error can't find the container with id 271aa8660ba901f534eef6334082d3956cbd4a961013d9c57e48f3b78a4b5573
Nov 02 15:49:20.532883 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:20.532860    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"]
Nov 02 15:49:21.421163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:21.421127    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj" event={"ID":"15d6fd17-fdd7-442d-a004-96d2c4e73a3b","Type":"ContainerStarted","Data":"0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f"}
Nov 02 15:49:21.421635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:21.421191    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj" event={"ID":"15d6fd17-fdd7-442d-a004-96d2c4e73a3b","Type":"ContainerStarted","Data":"271aa8660ba901f534eef6334082d3956cbd4a961013d9c57e48f3b78a4b5573"}
Nov 02 15:49:21.421635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:21.421415    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:21.636741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:21.636675    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj" podStartSLOduration=2.228269877 podStartE2EDuration="2.636653938s" totalImagesPullingTime="408.384061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:21.476183545 +0000 UTC m=+3213.734252997" watchObservedRunningTime="2026-11-02 15:49:21.636653938 +0000 UTC m=+3213.894723392"
Nov 02 15:49:21.638088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:21.638064    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"]
Nov 02 15:49:21.690626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:21.690531    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"]
Nov 02 15:49:26.638871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:26.638830    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx" podUID="279132d9-cd42-49ef-a4e4-f8623502ecc6" containerName="payload-processing" containerID="cri-o://b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844" gracePeriod=25
Nov 02 15:49:26.691560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:26.691526    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h" podUID="5ad2e454-f47f-4e96-af2b-ffd49da0adbd" containerName="payload-pre-processing" containerID="cri-o://58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b" gracePeriod=25
Nov 02 15:49:26.986451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:26.986419    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:49:27.015284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.015262    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:49:27.059503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.059472    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v\" (UniqueName: \"kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v\") pod \"279132d9-cd42-49ef-a4e4-f8623502ecc6\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") "
Nov 02 15:49:27.059663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.059533    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume\") pod \"279132d9-cd42-49ef-a4e4-f8623502ecc6\" (UID: \"279132d9-cd42-49ef-a4e4-f8623502ecc6\") "
Nov 02 15:49:27.060028 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.060002    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume" pod "279132d9-cd42-49ef-a4e4-f8623502ecc6" (UID: "279132d9-cd42-49ef-a4e4-f8623502ecc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:27.060237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.060177    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279132d9-cd42-49ef-a4e4-f8623502ecc6-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:27.061815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.061790    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v" pod "279132d9-cd42-49ef-a4e4-f8623502ecc6" (UID: "279132d9-cd42-49ef-a4e4-f8623502ecc6"). InnerVolumeSpecName "kube-api-access-tcp8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:27.161551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.161484    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume\") pod \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") "
Nov 02 15:49:27.161660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.161562    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb\" (UniqueName: \"kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb\") pod \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\" (UID: \"5ad2e454-f47f-4e96-af2b-ffd49da0adbd\") "
Nov 02 15:49:27.161921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.161897    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume" pod "5ad2e454-f47f-4e96-af2b-ffd49da0adbd" (UID: "5ad2e454-f47f-4e96-af2b-ffd49da0adbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:27.162094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.162077    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcp8v\" (UniqueName: \"kubernetes.io/projected/279132d9-cd42-49ef-a4e4-f8623502ecc6-kube-api-access-tcp8v\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:27.162128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.162097    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:27.163589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.163573    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb" pod "5ad2e454-f47f-4e96-af2b-ffd49da0adbd" (UID: "5ad2e454-f47f-4e96-af2b-ffd49da0adbd"). InnerVolumeSpecName "kube-api-access-fpnvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:27.262827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.262801    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnvb\" (UniqueName: \"kubernetes.io/projected/5ad2e454-f47f-4e96-af2b-ffd49da0adbd-kube-api-access-fpnvb\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:27.437128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.437103    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:49:27.475482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.475451    5041 generic.go:356] "Generic (PLEG): container finished" podID="279132d9-cd42-49ef-a4e4-f8623502ecc6" containerID="b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844" exitCode=0
Nov 02 15:49:27.475594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.475505    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"
Nov 02 15:49:27.475594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.475542    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx" event={"ID":"279132d9-cd42-49ef-a4e4-f8623502ecc6","Type":"ContainerDied","Data":"b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844"}
Nov 02 15:49:27.475594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.475591    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx" event={"ID":"279132d9-cd42-49ef-a4e4-f8623502ecc6","Type":"ContainerDied","Data":"27e83554ce4ec967cac2d895d24ace2dae9178d3d3bf75c1b931f2978071f5e1"}
Nov 02 15:49:27.475751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.475612    5041 scope.go:122] "RemoveContainer" containerID="b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844"
Nov 02 15:49:27.477247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.477226    5041 generic.go:356] "Generic (PLEG): container finished" podID="5ad2e454-f47f-4e96-af2b-ffd49da0adbd" containerID="58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b" exitCode=0
Nov 02 15:49:27.477353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.477274    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"
Nov 02 15:49:27.477449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.477338    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h" event={"ID":"5ad2e454-f47f-4e96-af2b-ffd49da0adbd","Type":"ContainerDied","Data":"58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b"}
Nov 02 15:49:27.477449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.477427    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h" event={"ID":"5ad2e454-f47f-4e96-af2b-ffd49da0adbd","Type":"ContainerDied","Data":"d8bee9ef9c311353031666b7526c7509c63188f96fc4b9c4e1018750a2bd0519"}
Nov 02 15:49:27.490082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.490058    5041 scope.go:122] "RemoveContainer" containerID="b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844"
Nov 02 15:49:27.490457 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:27.490433    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844\": container with ID starting with b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844 not found: ID does not exist" containerID="b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844"
Nov 02 15:49:27.490557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.490465    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844"} err="failed to get container status \"b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844\": rpc error: code = NotFound desc = could not find container \"b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844\": container with ID starting with b92292db903fb296e6f71ad4ae6ca1a1d4577e3df7af631864aebad156b5b844 not found: ID does not exist"
Nov 02 15:49:27.490557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.490497    5041 scope.go:122] "RemoveContainer" containerID="58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b"
Nov 02 15:49:27.505921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.505898    5041 scope.go:122] "RemoveContainer" containerID="58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b"
Nov 02 15:49:27.506238 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:27.506215    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b\": container with ID starting with 58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b not found: ID does not exist" containerID="58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b"
Nov 02 15:49:27.506322 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.506243    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b"} err="failed to get container status \"58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b\": rpc error: code = NotFound desc = could not find container \"58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b\": container with ID starting with 58bdbee53075e3d95ba7f6391cf076d84974794f1457b88567499dc1a2cbf21b not found: ID does not exist"
Nov 02 15:49:27.545271 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.545244    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"]
Nov 02 15:49:27.551436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.551411    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-869d9d4ccf-6999h"]
Nov 02 15:49:27.582614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.582592    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"]
Nov 02 15:49:27.587033 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.587010    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-c586d967-9vrzx"]
Nov 02 15:49:27.890638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.890569    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279132d9-cd42-49ef-a4e4-f8623502ecc6" path="/var/lib/kubelet/pods/279132d9-cd42-49ef-a4e4-f8623502ecc6/volumes"
Nov 02 15:49:27.890991 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:27.890964    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad2e454-f47f-4e96-af2b-ffd49da0adbd" path="/var/lib/kubelet/pods/5ad2e454-f47f-4e96-af2b-ffd49da0adbd/volumes"
Nov 02 15:49:31.619881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:31.619849    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"]
Nov 02 15:49:31.620285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:31.620035    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p" podUID="e5fcda3d-6d38-4b4a-8c64-f7724076da0a" containerName="payload-processing" containerID="cri-o://584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608" gracePeriod=30
Nov 02 15:49:31.919165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:31.919138    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:49:32.037385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.036993    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz\" (UniqueName: \"kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz\") pod \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") "
Nov 02 15:49:32.037385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.037062    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume\") pod \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\" (UID: \"e5fcda3d-6d38-4b4a-8c64-f7724076da0a\") "
Nov 02 15:49:32.042078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.041825    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume" pod "e5fcda3d-6d38-4b4a-8c64-f7724076da0a" (UID: "e5fcda3d-6d38-4b4a-8c64-f7724076da0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:32.053183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.053148    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz" pod "e5fcda3d-6d38-4b4a-8c64-f7724076da0a" (UID: "e5fcda3d-6d38-4b4a-8c64-f7724076da0a"). InnerVolumeSpecName "kube-api-access-29npz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:32.139196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.139123    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29npz\" (UniqueName: \"kubernetes.io/projected/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-kube-api-access-29npz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.139196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.139152    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fcda3d-6d38-4b4a-8c64-f7724076da0a-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.526313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.526277    5041 generic.go:356] "Generic (PLEG): container finished" podID="e5fcda3d-6d38-4b4a-8c64-f7724076da0a" containerID="584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608" exitCode=0
Nov 02 15:49:32.526520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.526318    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"
Nov 02 15:49:32.526520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.526388    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p" event={"ID":"e5fcda3d-6d38-4b4a-8c64-f7724076da0a","Type":"ContainerDied","Data":"584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608"}
Nov 02 15:49:32.526520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.526443    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p" event={"ID":"e5fcda3d-6d38-4b4a-8c64-f7724076da0a","Type":"ContainerDied","Data":"852cdba1cf82637753d2b9a8792c84622cb6a95769c1b45058b4efe739038680"}
Nov 02 15:49:32.526520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.526467    5041 scope.go:122] "RemoveContainer" containerID="584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608"
Nov 02 15:49:32.541604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.541585    5041 scope.go:122] "RemoveContainer" containerID="584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608"
Nov 02 15:49:32.541904 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:32.541883    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608\": container with ID starting with 584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608 not found: ID does not exist" containerID="584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608"
Nov 02 15:49:32.541979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.541910    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608"} err="failed to get container status \"584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608\": rpc error: code = NotFound desc = could not find container \"584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608\": container with ID starting with 584bd29b33249f497959662ae6ead4fce188aefac8ba0f194ddc6d44f1b03608 not found: ID does not exist"
Nov 02 15:49:32.580663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.580640    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"]
Nov 02 15:49:32.592035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:32.592014    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-24255b-c6d487764-8xh9p"]
Nov 02 15:49:33.890000 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:33.889964    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fcda3d-6d38-4b4a-8c64-f7724076da0a" path="/var/lib/kubelet/pods/e5fcda3d-6d38-4b4a-8c64-f7724076da0a/volumes"
Nov 02 15:49:36.769849 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:36.769813    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:49:38.316135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316101    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"]
Nov 02 15:49:38.316915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316900    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5ad2e454-f47f-4e96-af2b-ffd49da0adbd" containerName="payload-pre-processing"
Nov 02 15:49:38.316962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316916    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad2e454-f47f-4e96-af2b-ffd49da0adbd" containerName="payload-pre-processing"
Nov 02 15:49:38.316962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316939    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e5fcda3d-6d38-4b4a-8c64-f7724076da0a" containerName="payload-processing"
Nov 02 15:49:38.316962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316944    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fcda3d-6d38-4b4a-8c64-f7724076da0a" containerName="payload-processing"
Nov 02 15:49:38.316962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316954    5041 cpu_manager.go:414] "containerMap: removing container" podUID="279132d9-cd42-49ef-a4e4-f8623502ecc6" containerName="payload-processing"
Nov 02 15:49:38.316962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.316959    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279132d9-cd42-49ef-a4e4-f8623502ecc6" containerName="payload-processing"
Nov 02 15:49:38.317157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.317078    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad2e454-f47f-4e96-af2b-ffd49da0adbd" containerName="payload-pre-processing"
Nov 02 15:49:38.317157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.317087    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fcda3d-6d38-4b4a-8c64-f7724076da0a" containerName="payload-processing"
Nov 02 15:49:38.317157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.317095    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="279132d9-cd42-49ef-a4e4-f8623502ecc6" containerName="payload-processing"
Nov 02 15:49:38.322810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.322785    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.342632 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.342609    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-af33a6-serving-cert\""
Nov 02 15:49:38.349575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.349549    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"]
Nov 02 15:49:38.416533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.416501    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.416674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.416539    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrtx\" (UniqueName: \"kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.416674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.416651    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.416838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.416819    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.517767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.517728    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.517972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.517853    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.517972 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:38.517874    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-serving-cert: secret "maas-api-e2e-ipp-b-af33a6-serving-cert" not found
Nov 02 15:49:38.517972 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:38.517945    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls podName:6bd96673-cbd1-41f3-8e48-5a2ce59d48bd nodeName:}" failed. No retries permitted until 2026-11-02 15:49:39.017929836 +0000 UTC m=+3231.275999267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls") pod "maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" (UID: "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd") : secret "maas-api-e2e-ipp-b-af33a6-serving-cert" not found
Nov 02 15:49:38.518163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.518030    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.518163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.518061    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrtx\" (UniqueName: \"kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.518741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.518719    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.520478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.520452    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:38.545744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:38.545721    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrtx\" (UniqueName: \"kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:39.023415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:39.023383    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:39.025893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:39.025868    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:39.240036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:39.240004    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:39.545945 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:49:39.545911    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd96673_cbd1_41f3_8e48_5a2ce59d48bd.slice/crio-f8daf8b4f5a33461e25e099bbe8df3da0272b86d779432e5bd05516155732035 WatchSource:0}: Error finding container f8daf8b4f5a33461e25e099bbe8df3da0272b86d779432e5bd05516155732035: Status 404 returned error can't find the container with id f8daf8b4f5a33461e25e099bbe8df3da0272b86d779432e5bd05516155732035
Nov 02 15:49:39.546417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:39.546392    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"]
Nov 02 15:49:39.547600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:39.547579    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:49:39.585100 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:39.585072    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" event={"ID":"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd","Type":"ContainerStarted","Data":"f8daf8b4f5a33461e25e099bbe8df3da0272b86d779432e5bd05516155732035"}
Nov 02 15:49:40.594933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:40.594893    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" event={"ID":"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd","Type":"ContainerStarted","Data":"1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76"}
Nov 02 15:49:40.595341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:40.594956    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:40.641044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:40.640988    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" podStartSLOduration=2.292576966 podStartE2EDuration="2.64097199s" totalImagesPullingTime="348.395024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:40.637027714 +0000 UTC m=+3232.895097166" watchObservedRunningTime="2026-11-02 15:49:40.64097199 +0000 UTC m=+3232.899041440"
Nov 02 15:49:46.610014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:46.609982    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:49:51.237327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:51.237288    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"]
Nov 02 15:49:51.272782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:51.272752    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"]
Nov 02 15:49:52.610215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:52.610182    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"]
Nov 02 15:49:52.610826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:52.610476    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" podUID="3d0cce51-33f1-42ce-97bc-1a9829c2322e" containerName="maas-api" containerID="cri-o://29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91" gracePeriod=30
Nov 02 15:49:52.941469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:52.941446    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:49:53.024399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.017614    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\") pod \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") "
Nov 02 15:49:53.024399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.017692    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m\" (UniqueName: \"kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m\") pod \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") "
Nov 02 15:49:53.024399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.018099    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs\") pod \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") "
Nov 02 15:49:53.024399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.018319    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca\") pod \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\" (UID: \"3d0cce51-33f1-42ce-97bc-1a9829c2322e\") "
Nov 02 15:49:53.024399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.018796    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca" pod "3d0cce51-33f1-42ce-97bc-1a9829c2322e" (UID: "3d0cce51-33f1-42ce-97bc-1a9829c2322e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:53.037385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.034718    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0cce51-33f1-42ce-97bc-1a9829c2322e-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:53.090678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.089570    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls" pod "3d0cce51-33f1-42ce-97bc-1a9829c2322e" (UID: "3d0cce51-33f1-42ce-97bc-1a9829c2322e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:53.090678 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.089614    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m" pod "3d0cce51-33f1-42ce-97bc-1a9829c2322e" (UID: "3d0cce51-33f1-42ce-97bc-1a9829c2322e"). InnerVolumeSpecName "kube-api-access-ttr2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:53.098483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.097783    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs" pod "3d0cce51-33f1-42ce-97bc-1a9829c2322e" (UID: "3d0cce51-33f1-42ce-97bc-1a9829c2322e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:53.135587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.135555    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:53.135587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.135585    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0cce51-33f1-42ce-97bc-1a9829c2322e-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:53.135769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.135595    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttr2m\" (UniqueName: \"kubernetes.io/projected/3d0cce51-33f1-42ce-97bc-1a9829c2322e-kube-api-access-ttr2m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:49:53.719069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.719028    5041 generic.go:356] "Generic (PLEG): container finished" podID="3d0cce51-33f1-42ce-97bc-1a9829c2322e" containerID="29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91" exitCode=0
Nov 02 15:49:53.719492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.719082    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"
Nov 02 15:49:53.719492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.719126    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" event={"ID":"3d0cce51-33f1-42ce-97bc-1a9829c2322e","Type":"ContainerDied","Data":"29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91"}
Nov 02 15:49:53.719492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.719180    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg" event={"ID":"3d0cce51-33f1-42ce-97bc-1a9829c2322e","Type":"ContainerDied","Data":"1b1f4fd7f44aa92c794a61e012ba78e1bad097235bbdd39c8bcd87d7c674393c"}
Nov 02 15:49:53.719492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.719207    5041 scope.go:122] "RemoveContainer" containerID="29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91"
Nov 02 15:49:53.730503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.730486    5041 scope.go:122] "RemoveContainer" containerID="29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91"
Nov 02 15:49:53.730753 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:49:53.730732    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91\": container with ID starting with 29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91 not found: ID does not exist" containerID="29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91"
Nov 02 15:49:53.730825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.730759    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91"} err="failed to get container status \"29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91\": rpc error: code = NotFound desc = could not find container \"29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91\": container with ID starting with 29473c89d071e46ca4d348915a0142e240c5370c2269d96995cebc59267d8e91 not found: ID does not exist"
Nov 02 15:49:53.770737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.770706    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"]
Nov 02 15:49:53.776229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.776208    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-246137-6d594bc6b4-44rdg"]
Nov 02 15:49:53.890022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:53.889992    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0cce51-33f1-42ce-97bc-1a9829c2322e" path="/var/lib/kubelet/pods/3d0cce51-33f1-42ce-97bc-1a9829c2322e/volumes"
Nov 02 15:49:56.237844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:56.237760    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerName="payload-processing" containerID="cri-o://158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121" gracePeriod=25
Nov 02 15:49:56.273482 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:56.273441    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" containerName="payload-pre-processing" containerID="cri-o://07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48" gracePeriod=25
Nov 02 15:49:59.507597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:49:59.507558    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:00.388906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.388870    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"]
Nov 02 15:50:00.389964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.389942    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3d0cce51-33f1-42ce-97bc-1a9829c2322e" containerName="maas-api"
Nov 02 15:50:00.390034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.389967    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0cce51-33f1-42ce-97bc-1a9829c2322e" containerName="maas-api"
Nov 02 15:50:00.390131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.390121    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0cce51-33f1-42ce-97bc-1a9829c2322e" containerName="maas-api"
Nov 02 15:50:00.395707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.395688    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.415176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.415147    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"]
Nov 02 15:50:00.423406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.421014    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5721b1-dockercfg-ktpvb\""
Nov 02 15:50:00.423406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.421102    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5721b1\""
Nov 02 15:50:00.523177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.523139    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.523601 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.523407    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.538890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.538860    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:00.566771 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.566747    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"]
Nov 02 15:50:00.571435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.571419    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.593970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.593945    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"]
Nov 02 15:50:00.624639 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.624608    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.624798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.624754    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.626003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.625977    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.647830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.647762    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k\") pod \"payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.710325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.710294    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:00.726527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.726483    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.726689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.726668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qst\" (UniqueName: \"kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst\") pod \"payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.829448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.828149    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.829448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.828297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qst\" (UniqueName: \"kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst\") pod \"payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.829448 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.829381    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.863826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.863798    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qst\" (UniqueName: \"kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst\") pod \"payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:00.886316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:00.886285    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:01.043034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.042029    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"]
Nov 02 15:50:01.196204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.196167    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"]
Nov 02 15:50:01.795109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.795074    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2" event={"ID":"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3","Type":"ContainerStarted","Data":"a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290"}
Nov 02 15:50:01.795610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.795133    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2" event={"ID":"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3","Type":"ContainerStarted","Data":"608f2465bfdb9d81dde5b101fa4d89455786b807499a45712ac5d2a47b18020b"}
Nov 02 15:50:01.795610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.795175    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:01.797051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.797020    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8" event={"ID":"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9","Type":"ContainerStarted","Data":"a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87"}
Nov 02 15:50:01.797199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.797067    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8" event={"ID":"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9","Type":"ContainerStarted","Data":"83aafa07b9c1cf9afb64b411b76b0727f0e33369d12aed00b987c18dd6cc67d5"}
Nov 02 15:50:01.797199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.797090    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:01.839310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.839268    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2" podStartSLOduration=1.839255574 podStartE2EDuration="1.839255574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:01.835167312 +0000 UTC m=+3254.093236800" watchObservedRunningTime="2026-11-02 15:50:01.839255574 +0000 UTC m=+3254.097325023"
Nov 02 15:50:01.875851 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:01.875807    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8" podStartSLOduration=1.875794903 podStartE2EDuration="1.875794903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:01.874282372 +0000 UTC m=+3254.132351822" watchObservedRunningTime="2026-11-02 15:50:01.875794903 +0000 UTC m=+3254.133864353"
Nov 02 15:50:06.770566 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:06.770528    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:50:07.476073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:07.476039    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"]
Nov 02 15:50:09.507174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:09.507130    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:10.539060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:10.539022    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:11.899673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:11.899633    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"]
Nov 02 15:50:12.477097 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.477059    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q" podUID="fd404e47-5b89-4bcf-875a-47e326f9ee38" containerName="payload-pre-processing" containerID="cri-o://b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1" gracePeriod=25
Nov 02 15:50:12.778695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.778669    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:50:12.808010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.807976    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:50:12.808287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.808265    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:50:12.880257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.880225    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume\") pod \"fd404e47-5b89-4bcf-875a-47e326f9ee38\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") "
Nov 02 15:50:12.880504 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.880484    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb\" (UniqueName: \"kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb\") pod \"fd404e47-5b89-4bcf-875a-47e326f9ee38\" (UID: \"fd404e47-5b89-4bcf-875a-47e326f9ee38\") "
Nov 02 15:50:12.880699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.880651    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume" pod "fd404e47-5b89-4bcf-875a-47e326f9ee38" (UID: "fd404e47-5b89-4bcf-875a-47e326f9ee38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:12.881296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.881270    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd404e47-5b89-4bcf-875a-47e326f9ee38-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:12.887624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.887596    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb" pod "fd404e47-5b89-4bcf-875a-47e326f9ee38" (UID: "fd404e47-5b89-4bcf-875a-47e326f9ee38"). InnerVolumeSpecName "kube-api-access-fwwpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:12.895136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.895106    5041 generic.go:356] "Generic (PLEG): container finished" podID="fd404e47-5b89-4bcf-875a-47e326f9ee38" containerID="b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1" exitCode=0
Nov 02 15:50:12.895232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.895144    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"
Nov 02 15:50:12.895232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.895219    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q" event={"ID":"fd404e47-5b89-4bcf-875a-47e326f9ee38","Type":"ContainerDied","Data":"b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1"}
Nov 02 15:50:12.895418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.895256    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q" event={"ID":"fd404e47-5b89-4bcf-875a-47e326f9ee38","Type":"ContainerDied","Data":"f31a633fab018ac2044969e2ae859f0fd4629bdc72d9bb1351febc2077dca2cd"}
Nov 02 15:50:12.895418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.895284    5041 scope.go:122] "RemoveContainer" containerID="b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1"
Nov 02 15:50:12.925279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.925179    5041 scope.go:122] "RemoveContainer" containerID="b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1"
Nov 02 15:50:12.925692 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:12.925608    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1\": container with ID starting with b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1 not found: ID does not exist" containerID="b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1"
Nov 02 15:50:12.925692 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.925647    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1"} err="failed to get container status \"b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1\": rpc error: code = NotFound desc = could not find container \"b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1\": container with ID starting with b34d5aed6234cc387ededa66b27d0e4b9b770a39ca63e88271f099db4899fbb1 not found: ID does not exist"
Nov 02 15:50:12.964470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.964445    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"]
Nov 02 15:50:12.970641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.970617    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-d4b46dbbc-j269q"]
Nov 02 15:50:12.982336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:12.982309    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwpb\" (UniqueName: \"kubernetes.io/projected/fd404e47-5b89-4bcf-875a-47e326f9ee38-kube-api-access-fwwpb\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:13.889586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:13.889557    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd404e47-5b89-4bcf-875a-47e326f9ee38" path="/var/lib/kubelet/pods/fd404e47-5b89-4bcf-875a-47e326f9ee38/volumes"
Nov 02 15:50:14.476229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.476196    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"]
Nov 02 15:50:14.477052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.477036    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fd404e47-5b89-4bcf-875a-47e326f9ee38" containerName="payload-pre-processing"
Nov 02 15:50:14.477111 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.477055    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd404e47-5b89-4bcf-875a-47e326f9ee38" containerName="payload-pre-processing"
Nov 02 15:50:14.477231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.477222    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd404e47-5b89-4bcf-875a-47e326f9ee38" containerName="payload-pre-processing"
Nov 02 15:50:14.482255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.482238    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.501058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.501023    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.501269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.501248    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgcw\" (UniqueName: \"kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.503431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.503407    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"]
Nov 02 15:50:14.509420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.509400    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a433eb\""
Nov 02 15:50:14.509537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.509501    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a433eb-dockercfg-rw7z8\""
Nov 02 15:50:14.593836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.593775    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"]
Nov 02 15:50:14.599087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.599062    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.610790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.603733    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.610790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.604900    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgcw\" (UniqueName: \"kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.610790 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.604632    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.626512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.625402    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"]
Nov 02 15:50:14.647313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.647278    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgcw\" (UniqueName: \"kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.706309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.706266    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.706597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.706574    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfwm\" (UniqueName: \"kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm\") pod \"payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.797756 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.797680    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:14.807441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.807416    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.807629 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.807607    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfwm\" (UniqueName: \"kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm\") pod \"payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.808307 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.808279    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.831865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.831831    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfwm\" (UniqueName: \"kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm\") pod \"payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:14.952106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:14.952070    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:15.115629 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:50:15.115581    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0832fb1c_8119_4840_afbf_9f7aac288a51.slice/crio-7cb8a0359afa819f165b51b277b175616791d5538f72d635424e0d783c603a81 WatchSource:0}: Error finding container 7cb8a0359afa819f165b51b277b175616791d5538f72d635424e0d783c603a81: Status 404 returned error can't find the container with id 7cb8a0359afa819f165b51b277b175616791d5538f72d635424e0d783c603a81
Nov 02 15:50:15.120707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.120660    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"]
Nov 02 15:50:15.267709 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.267679    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"]
Nov 02 15:50:15.268139 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:50:15.268113    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd880b44a_6a89_4e74_bc4c_b17f955f7cf6.slice/crio-fbef3c8562f3ab142d0d0efa8a950c4c734ff4854e3c2f2c98ce10caa850e1d4 WatchSource:0}: Error finding container fbef3c8562f3ab142d0d0efa8a950c4c734ff4854e3c2f2c98ce10caa850e1d4: Status 404 returned error can't find the container with id fbef3c8562f3ab142d0d0efa8a950c4c734ff4854e3c2f2c98ce10caa850e1d4
Nov 02 15:50:15.920088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.920046    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6" event={"ID":"d880b44a-6a89-4e74-bc4c-b17f955f7cf6","Type":"ContainerStarted","Data":"7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111"}
Nov 02 15:50:15.920518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.920107    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6" event={"ID":"d880b44a-6a89-4e74-bc4c-b17f955f7cf6","Type":"ContainerStarted","Data":"fbef3c8562f3ab142d0d0efa8a950c4c734ff4854e3c2f2c98ce10caa850e1d4"}
Nov 02 15:50:15.920518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.920171    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:15.921764 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.921734    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn" event={"ID":"0832fb1c-8119-4840-afbf-9f7aac288a51","Type":"ContainerStarted","Data":"0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac"}
Nov 02 15:50:15.921867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.921774    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn" event={"ID":"0832fb1c-8119-4840-afbf-9f7aac288a51","Type":"ContainerStarted","Data":"7cb8a0359afa819f165b51b277b175616791d5538f72d635424e0d783c603a81"}
Nov 02 15:50:15.921867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.921811    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:15.965029 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:15.964973    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6" podStartSLOduration=1.964962144 podStartE2EDuration="1.964962144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:15.960320748 +0000 UTC m=+3268.218390198" watchObservedRunningTime="2026-11-02 15:50:15.964962144 +0000 UTC m=+3268.223031594"
Nov 02 15:50:16.003578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.003527    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn" podStartSLOduration=2.003513323 podStartE2EDuration="2.003513323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:16.000299955 +0000 UTC m=+3268.258369434" watchObservedRunningTime="2026-11-02 15:50:16.003513323 +0000 UTC m=+3268.261582773"
Nov 02 15:50:16.586626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.586598    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:50:16.590352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.590331    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:50:16.638152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.638123    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume\") pod \"60e84e50-c665-4c63-9744-106f9c0dc549\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") "
Nov 02 15:50:16.638268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.638159    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6\" (UniqueName: \"kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6\") pod \"60e84e50-c665-4c63-9744-106f9c0dc549\" (UID: \"60e84e50-c665-4c63-9744-106f9c0dc549\") "
Nov 02 15:50:16.638346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.638309    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume\") pod \"bd42e35e-24c6-4644-a98a-a790e91652eb\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") "
Nov 02 15:50:16.638509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.638430    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4\" (UniqueName: \"kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4\") pod \"bd42e35e-24c6-4644-a98a-a790e91652eb\" (UID: \"bd42e35e-24c6-4644-a98a-a790e91652eb\") "
Nov 02 15:50:16.638822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.638537    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume" pod "60e84e50-c665-4c63-9744-106f9c0dc549" (UID: "60e84e50-c665-4c63-9744-106f9c0dc549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:16.638903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.638836    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume" pod "bd42e35e-24c6-4644-a98a-a790e91652eb" (UID: "bd42e35e-24c6-4644-a98a-a790e91652eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:16.639074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.639053    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e84e50-c665-4c63-9744-106f9c0dc549-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:16.639074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.639072    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd42e35e-24c6-4644-a98a-a790e91652eb-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:16.640939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.640913    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6" pod "60e84e50-c665-4c63-9744-106f9c0dc549" (UID: "60e84e50-c665-4c63-9744-106f9c0dc549"). InnerVolumeSpecName "kube-api-access-94bl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:16.641047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.641028    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4" pod "bd42e35e-24c6-4644-a98a-a790e91652eb" (UID: "bd42e35e-24c6-4644-a98a-a790e91652eb"). InnerVolumeSpecName "kube-api-access-ggkq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:16.740689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.740656    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkq4\" (UniqueName: \"kubernetes.io/projected/bd42e35e-24c6-4644-a98a-a790e91652eb-kube-api-access-ggkq4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:16.740689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.740689    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bl6\" (UniqueName: \"kubernetes.io/projected/60e84e50-c665-4c63-9744-106f9c0dc549-kube-api-access-94bl6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:16.900768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.900647    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" podUID="1332eff4-31e9-43b3-9454-413da38177f0" containerName="payload-processing" containerID="cri-o://7d4610bd9d879feb7ed134689ace04c27f24668fd57782de878fa29834a16125" gracePeriod=25
Nov 02 15:50:16.931915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.931883    5041 generic.go:356] "Generic (PLEG): container finished" podID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerID="158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121" exitCode=0
Nov 02 15:50:16.932310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.931925    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"
Nov 02 15:50:16.932310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.931923    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" event={"ID":"bd42e35e-24c6-4644-a98a-a790e91652eb","Type":"ContainerDied","Data":"158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121"}
Nov 02 15:50:16.932310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.932054    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j" event={"ID":"bd42e35e-24c6-4644-a98a-a790e91652eb","Type":"ContainerDied","Data":"2a8be6369681b63c0da02823d0df4a43285f1df124b803901f5d047d102f0dd2"}
Nov 02 15:50:16.932310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.932089    5041 scope.go:122] "RemoveContainer" containerID="158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121"
Nov 02 15:50:16.934795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.934760    5041 generic.go:356] "Generic (PLEG): container finished" podID="60e84e50-c665-4c63-9744-106f9c0dc549" containerID="07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48" exitCode=0
Nov 02 15:50:16.934889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.934808    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"
Nov 02 15:50:16.934951 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.934896    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" event={"ID":"60e84e50-c665-4c63-9744-106f9c0dc549","Type":"ContainerDied","Data":"07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48"}
Nov 02 15:50:16.935018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.934951    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n" event={"ID":"60e84e50-c665-4c63-9744-106f9c0dc549","Type":"ContainerDied","Data":"d0b936303fefd38db88697b2ca745fbb1d456901aba82961bbd78b75da18f5f6"}
Nov 02 15:50:16.949175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.949154    5041 scope.go:122] "RemoveContainer" containerID="158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121"
Nov 02 15:50:16.949558 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:16.949523    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121\": container with ID starting with 158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121 not found: ID does not exist" containerID="158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121"
Nov 02 15:50:16.949725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.949561    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121"} err="failed to get container status \"158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121\": rpc error: code = NotFound desc = could not find container \"158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121\": container with ID starting with 158e7808ddba4c07dbfac60eaab2a879dbe8966cfbae00ae5e99aec625351121 not found: ID does not exist"
Nov 02 15:50:16.949725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.949582    5041 scope.go:122] "RemoveContainer" containerID="07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48"
Nov 02 15:50:16.961238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.961221    5041 scope.go:122] "RemoveContainer" containerID="07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48"
Nov 02 15:50:16.961524 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:16.961504    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48\": container with ID starting with 07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48 not found: ID does not exist" containerID="07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48"
Nov 02 15:50:16.961619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.961527    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48"} err="failed to get container status \"07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48\": rpc error: code = NotFound desc = could not find container \"07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48\": container with ID starting with 07e013c454f707c7432d45364178f5a0cf2e5a8bfaf6102ef5d57e1a14870d48 not found: ID does not exist"
Nov 02 15:50:16.985281 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.985256    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"]
Nov 02 15:50:16.994434 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:16.994411    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-246137-7497cfc4bf-nr57j"]
Nov 02 15:50:17.048563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:17.048530    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"]
Nov 02 15:50:17.053239 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:17.053213    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-246137-7d768bb58c-ggj8n"]
Nov 02 15:50:17.902223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:17.902185    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" path="/var/lib/kubelet/pods/60e84e50-c665-4c63-9744-106f9c0dc549/volumes"
Nov 02 15:50:17.902636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:17.902621    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" path="/var/lib/kubelet/pods/bd42e35e-24c6-4644-a98a-a790e91652eb/volumes"
Nov 02 15:50:23.090681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:23.090645    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" podUID="1332eff4-31e9-43b3-9454-413da38177f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:24.324785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:24.324749    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"]
Nov 02 15:50:26.938476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:26.938443    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:50:26.938898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:26.938516    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:50:29.325703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.325585    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl" podUID="7234bcbd-9e17-46b9-9407-207d8102ff30" containerName="payload-pre-processing" containerID="cri-o://de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d" gracePeriod=25
Nov 02 15:50:29.624613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.624585    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:50:29.793011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.792981    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume\") pod \"7234bcbd-9e17-46b9-9407-207d8102ff30\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") "
Nov 02 15:50:29.793215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.793108    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw\" (UniqueName: \"kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw\") pod \"7234bcbd-9e17-46b9-9407-207d8102ff30\" (UID: \"7234bcbd-9e17-46b9-9407-207d8102ff30\") "
Nov 02 15:50:29.793441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.793416    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume" pod "7234bcbd-9e17-46b9-9407-207d8102ff30" (UID: "7234bcbd-9e17-46b9-9407-207d8102ff30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:29.795220 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.795199    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw" pod "7234bcbd-9e17-46b9-9407-207d8102ff30" (UID: "7234bcbd-9e17-46b9-9407-207d8102ff30"). InnerVolumeSpecName "kube-api-access-dvwgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:29.894200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.894145    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwgw\" (UniqueName: \"kubernetes.io/projected/7234bcbd-9e17-46b9-9407-207d8102ff30-kube-api-access-dvwgw\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:29.894200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:29.894171    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7234bcbd-9e17-46b9-9407-207d8102ff30-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:30.051287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.051257    5041 generic.go:356] "Generic (PLEG): container finished" podID="7234bcbd-9e17-46b9-9407-207d8102ff30" containerID="de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d" exitCode=0
Nov 02 15:50:30.051483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.051295    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"
Nov 02 15:50:30.051483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.051359    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl" event={"ID":"7234bcbd-9e17-46b9-9407-207d8102ff30","Type":"ContainerDied","Data":"de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d"}
Nov 02 15:50:30.051483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.051435    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl" event={"ID":"7234bcbd-9e17-46b9-9407-207d8102ff30","Type":"ContainerDied","Data":"ad4b5cb424e6f1ca087c13c4950bf69a666ea9a8f54eac6768f4830c4252eb43"}
Nov 02 15:50:30.051682 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.051464    5041 scope.go:122] "RemoveContainer" containerID="de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d"
Nov 02 15:50:30.062734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.062717    5041 scope.go:122] "RemoveContainer" containerID="de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d"
Nov 02 15:50:30.063004 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:30.062981    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d\": container with ID starting with de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d not found: ID does not exist" containerID="de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d"
Nov 02 15:50:30.063054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.063011    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d"} err="failed to get container status \"de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d\": rpc error: code = NotFound desc = could not find container \"de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d\": container with ID starting with de2c8b0fceb5bd5e62191893226113b9f6ca0c09895dc660276268ec9a68a43d not found: ID does not exist"
Nov 02 15:50:30.104333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.104306    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"]
Nov 02 15:50:30.108550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:30.108527    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-f5567f9bc-2rvkl"]
Nov 02 15:50:31.888849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:31.888813    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7234bcbd-9e17-46b9-9407-207d8102ff30" path="/var/lib/kubelet/pods/7234bcbd-9e17-46b9-9407-207d8102ff30/volumes"
Nov 02 15:50:33.090333 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:33.090298    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" podUID="1332eff4-31e9-43b3-9454-413da38177f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:36.769548 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:36.769510    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:50:37.110936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.110897    5041 generic.go:356] "Generic (PLEG): container finished" podID="1332eff4-31e9-43b3-9454-413da38177f0" containerID="7d4610bd9d879feb7ed134689ace04c27f24668fd57782de878fa29834a16125" exitCode=0
Nov 02 15:50:37.111128 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.110991    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" event={"ID":"1332eff4-31e9-43b3-9454-413da38177f0","Type":"ContainerDied","Data":"7d4610bd9d879feb7ed134689ace04c27f24668fd57782de878fa29834a16125"}
Nov 02 15:50:37.185155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.185129    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:50:37.282836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.282764    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25\" (UniqueName: \"kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25\") pod \"1332eff4-31e9-43b3-9454-413da38177f0\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") "
Nov 02 15:50:37.282985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.282865    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume\") pod \"1332eff4-31e9-43b3-9454-413da38177f0\" (UID: \"1332eff4-31e9-43b3-9454-413da38177f0\") "
Nov 02 15:50:37.283309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.283278    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume" pod "1332eff4-31e9-43b3-9454-413da38177f0" (UID: "1332eff4-31e9-43b3-9454-413da38177f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:37.283459 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.283442    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1332eff4-31e9-43b3-9454-413da38177f0-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:37.284961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.284939    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25" pod "1332eff4-31e9-43b3-9454-413da38177f0" (UID: "1332eff4-31e9-43b3-9454-413da38177f0"). InnerVolumeSpecName "kube-api-access-zrd25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:37.384527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:37.384496    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrd25\" (UniqueName: \"kubernetes.io/projected/1332eff4-31e9-43b3-9454-413da38177f0-kube-api-access-zrd25\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:38.120186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:38.120151    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h" event={"ID":"1332eff4-31e9-43b3-9454-413da38177f0","Type":"ContainerDied","Data":"ffca5ef0342881e5739c66a73ef0373228a5574b373737c51254dc7b5989b63b"}
Nov 02 15:50:38.120721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:38.120197    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"
Nov 02 15:50:38.120721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:38.120220    5041 scope.go:122] "RemoveContainer" containerID="7d4610bd9d879feb7ed134689ace04c27f24668fd57782de878fa29834a16125"
Nov 02 15:50:38.173324 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:38.173295    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"]
Nov 02 15:50:38.184717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:38.184687    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0a7163-66d44ff89c-nwk9h"]
Nov 02 15:50:39.889460 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:39.889427    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1332eff4-31e9-43b3-9454-413da38177f0" path="/var/lib/kubelet/pods/1332eff4-31e9-43b3-9454-413da38177f0/volumes"
Nov 02 15:50:41.952715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.952670    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"]
Nov 02 15:50:41.953481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953465    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerName="payload-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953482    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerName="payload-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953503    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7234bcbd-9e17-46b9-9407-207d8102ff30" containerName="payload-pre-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953508    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7234bcbd-9e17-46b9-9407-207d8102ff30" containerName="payload-pre-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953529    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1332eff4-31e9-43b3-9454-413da38177f0" containerName="payload-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953534    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1332eff4-31e9-43b3-9454-413da38177f0" containerName="payload-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953546    5041 cpu_manager.go:414] "containerMap: removing container" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" containerName="payload-pre-processing"
Nov 02 15:50:41.953556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953551    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" containerName="payload-pre-processing"
Nov 02 15:50:41.953781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953675    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd42e35e-24c6-4644-a98a-a790e91652eb" containerName="payload-processing"
Nov 02 15:50:41.953781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953687    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7234bcbd-9e17-46b9-9407-207d8102ff30" containerName="payload-pre-processing"
Nov 02 15:50:41.953781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953696    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1332eff4-31e9-43b3-9454-413da38177f0" containerName="payload-processing"
Nov 02 15:50:41.953781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.953704    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e84e50-c665-4c63-9744-106f9c0dc549" containerName="payload-pre-processing"
Nov 02 15:50:41.959004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.958981    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:41.974192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.974158    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8537e9-dockercfg-7gwxc\""
Nov 02 15:50:41.974441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.974422    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8537e9\""
Nov 02 15:50:41.979530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:41.979507    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"]
Nov 02 15:50:42.037374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.037322    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrl47\" (UniqueName: \"kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.037556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.037532    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.122445 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.122407    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"]
Nov 02 15:50:42.127676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.127655    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.138800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.138777    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrl47\" (UniqueName: \"kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.138941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.138925    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.139614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.139596    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.150294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.150269    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"]
Nov 02 15:50:42.162547 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.162523    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrl47\" (UniqueName: \"kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47\") pod \"payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.240469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.240404    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.240577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.240525    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27pz\" (UniqueName: \"kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz\") pod \"payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.274641 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.274615    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:42.342059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.342016    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27pz\" (UniqueName: \"kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz\") pod \"payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.342330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.342287    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.343158 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.343128    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.367839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.367806    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27pz\" (UniqueName: \"kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz\") pod \"payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.442439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.442404    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:42.587948 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:50:42.587912    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b63d7_ab28_495d_8a45_45379a3891e1.slice/crio-78cccabadbce04a8062bf65201928e003436a0b38551ae42de568dcebfb54086 WatchSource:0}: Error finding container 78cccabadbce04a8062bf65201928e003436a0b38551ae42de568dcebfb54086: Status 404 returned error can't find the container with id 78cccabadbce04a8062bf65201928e003436a0b38551ae42de568dcebfb54086
Nov 02 15:50:42.590627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.590596    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"]
Nov 02 15:50:42.749768 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:50:42.749733    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22dfaed_e76a_4b46_b522_9f4a6d119018.slice/crio-62fc448f83cc14cf80954235aee636277d2e1d9642fed4bd660a0960eedd3153 WatchSource:0}: Error finding container 62fc448f83cc14cf80954235aee636277d2e1d9642fed4bd660a0960eedd3153: Status 404 returned error can't find the container with id 62fc448f83cc14cf80954235aee636277d2e1d9642fed4bd660a0960eedd3153
Nov 02 15:50:42.749929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:42.749790    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"]
Nov 02 15:50:43.165557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.165482    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph" event={"ID":"1d0b63d7-ab28-495d-8a45-45379a3891e1","Type":"ContainerStarted","Data":"2941c40016c7f33958953876f046812be5b70f07f1ffbd6e2ca3a3856830b49d"}
Nov 02 15:50:43.165557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.165543    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph" event={"ID":"1d0b63d7-ab28-495d-8a45-45379a3891e1","Type":"ContainerStarted","Data":"78cccabadbce04a8062bf65201928e003436a0b38551ae42de568dcebfb54086"}
Nov 02 15:50:43.166063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.165572    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:43.167339 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.167313    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp" event={"ID":"c22dfaed-e76a-4b46-b522-9f4a6d119018","Type":"ContainerStarted","Data":"7a60564fa42616c65e96552f7b8d651f439ff6c5c3ace90e111b0db7b64e0903"}
Nov 02 15:50:43.167474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.167352    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp" event={"ID":"c22dfaed-e76a-4b46-b522-9f4a6d119018","Type":"ContainerStarted","Data":"62fc448f83cc14cf80954235aee636277d2e1d9642fed4bd660a0960eedd3153"}
Nov 02 15:50:43.167474 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.167408    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:43.217489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.217446    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph" podStartSLOduration=2.217432539 podStartE2EDuration="2.217432539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:43.212635213 +0000 UTC m=+3295.470704693" watchObservedRunningTime="2026-11-02 15:50:43.217432539 +0000 UTC m=+3295.475501989"
Nov 02 15:50:43.253522 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:43.253483    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp" podStartSLOduration=1.253471602 podStartE2EDuration="1.253471602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:43.252233782 +0000 UTC m=+3295.510303232" watchObservedRunningTime="2026-11-02 15:50:43.253471602 +0000 UTC m=+3295.511541051"
Nov 02 15:50:49.321418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.321282    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:50:49.329856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.329831    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:50:49.604119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.604040    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"]
Nov 02 15:50:49.608781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.608763    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.628765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.628652    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a433eb-serving-cert\""
Nov 02 15:50:49.646659 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.646635    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"]
Nov 02 15:50:49.730104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.730076    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjd4x\" (UniqueName: \"kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.730261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.730127    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.730261 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.730204    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.730419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.730399    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.831600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.831572    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.831773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.831615    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.831773 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.831759    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.831870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.831815    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjd4x\" (UniqueName: \"kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.832585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.832530    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.834274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.834245    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.834412 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.834296    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.854477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.854405    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjd4x\" (UniqueName: \"kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x\") pod \"maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:49.925253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:49.925223    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:50.235258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:50.235226    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"]
Nov 02 15:50:50.236761 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:50:50.236737    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962bcb86_ff57_4c1f_a3f9_7fe06f4a0f65.slice/crio-3dcacf7cd1797007532235829fe5a88e21a76e8d0475a7f09420e18d7196fa0f WatchSource:0}: Error finding container 3dcacf7cd1797007532235829fe5a88e21a76e8d0475a7f09420e18d7196fa0f: Status 404 returned error can't find the container with id 3dcacf7cd1797007532235829fe5a88e21a76e8d0475a7f09420e18d7196fa0f
Nov 02 15:50:51.236076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:51.236031    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5" event={"ID":"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65","Type":"ContainerStarted","Data":"95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5"}
Nov 02 15:50:51.236534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:51.236106    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5" event={"ID":"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65","Type":"ContainerStarted","Data":"3dcacf7cd1797007532235829fe5a88e21a76e8d0475a7f09420e18d7196fa0f"}
Nov 02 15:50:51.236534 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:51.236136    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:51.281679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:51.281633    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5" podStartSLOduration=1.835760435 podStartE2EDuration="2.28162028s" totalImagesPullingTime="445.859845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:51.27645641 +0000 UTC m=+3303.534525863" watchObservedRunningTime="2026-11-02 15:50:51.28162028 +0000 UTC m=+3303.539689754"
Nov 02 15:50:54.176919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:54.176892    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:50:54.177421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:54.177296    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:50:55.912502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:55.912466    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"]
Nov 02 15:50:55.913007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:55.912703    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" podUID="58285986-2b9a-4804-9494-5c61afe725d7" containerName="maas-api" containerID="cri-o://e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323" gracePeriod=30
Nov 02 15:50:56.214971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.214942    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:50:56.287454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.287415    5041 generic.go:356] "Generic (PLEG): container finished" podID="58285986-2b9a-4804-9494-5c61afe725d7" containerID="e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323" exitCode=0
Nov 02 15:50:56.287616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.287461    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"
Nov 02 15:50:56.287616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.287506    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" event={"ID":"58285986-2b9a-4804-9494-5c61afe725d7","Type":"ContainerDied","Data":"e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323"}
Nov 02 15:50:56.287616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.287578    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf" event={"ID":"58285986-2b9a-4804-9494-5c61afe725d7","Type":"ContainerDied","Data":"14e61fe3fe178af7e4ffa60b1c6b3280411afc32b78c3790f85b86889d868c92"}
Nov 02 15:50:56.287616 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.287611    5041 scope.go:122] "RemoveContainer" containerID="e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323"
Nov 02 15:50:56.308982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.308961    5041 scope.go:122] "RemoveContainer" containerID="e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323"
Nov 02 15:50:56.309515 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:50:56.309466    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323\": container with ID starting with e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323 not found: ID does not exist" containerID="e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323"
Nov 02 15:50:56.309515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.309504    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323"} err="failed to get container status \"e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323\": rpc error: code = NotFound desc = could not find container \"e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323\": container with ID starting with e227ba571eb9c49ecbc1334b8232f19736709d825d78c0e7284e52e4fa0d1323 not found: ID does not exist"
Nov 02 15:50:56.313065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.313039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m\" (UniqueName: \"kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m\") pod \"58285986-2b9a-4804-9494-5c61afe725d7\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") "
Nov 02 15:50:56.315024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.314997    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs\") pod \"58285986-2b9a-4804-9494-5c61afe725d7\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") "
Nov 02 15:50:56.315131 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.315090    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m" pod "58285986-2b9a-4804-9494-5c61afe725d7" (UID: "58285986-2b9a-4804-9494-5c61afe725d7"). InnerVolumeSpecName "kube-api-access-gjc7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:56.315228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.315211    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca\") pod \"58285986-2b9a-4804-9494-5c61afe725d7\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") "
Nov 02 15:50:56.315283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.315260    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\") pod \"58285986-2b9a-4804-9494-5c61afe725d7\" (UID: \"58285986-2b9a-4804-9494-5c61afe725d7\") "
Nov 02 15:50:56.315646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.315624    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca" pod "58285986-2b9a-4804-9494-5c61afe725d7" (UID: "58285986-2b9a-4804-9494-5c61afe725d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:56.315897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.315876    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjc7m\" (UniqueName: \"kubernetes.io/projected/58285986-2b9a-4804-9494-5c61afe725d7-kube-api-access-gjc7m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.315995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.315898    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58285986-2b9a-4804-9494-5c61afe725d7-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.316921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.316900    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs" pod "58285986-2b9a-4804-9494-5c61afe725d7" (UID: "58285986-2b9a-4804-9494-5c61afe725d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:50:56.317195 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.317173    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls" pod "58285986-2b9a-4804-9494-5c61afe725d7" (UID: "58285986-2b9a-4804-9494-5c61afe725d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:50:56.430839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.430571    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.430839 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.430613    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58285986-2b9a-4804-9494-5c61afe725d7-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.647929 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.647889    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"]
Nov 02 15:50:56.654798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:56.654773    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a55a55-58d6b55686-pvcrf"]
Nov 02 15:50:57.249560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:57.249529    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:50:57.895451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:50:57.895413    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58285986-2b9a-4804-9494-5c61afe725d7" path="/var/lib/kubelet/pods/58285986-2b9a-4804-9494-5c61afe725d7/volumes"
Nov 02 15:51:03.392613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:03.392579    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"]
Nov 02 15:51:06.769767 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:06.769722    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:51:09.186577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:09.186542    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"]
Nov 02 15:51:09.222224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:09.222192    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"]
Nov 02 15:51:14.187782 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:14.187737    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" podUID="c9c52bec-5157-4483-9df4-413a53514093" containerName="payload-processing" containerID="cri-o://23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6" gracePeriod=25
Nov 02 15:51:14.222895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:14.222859    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing" containerID="cri-o://60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458" gracePeriod=25
Nov 02 15:51:15.674309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:15.674272    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"]
Nov 02 15:51:15.675130 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:15.674522    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" podUID="9488db94-ed3d-4d85-af40-5c77e36c5332" containerName="maas-api" containerID="cri-o://ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af" gracePeriod=30
Nov 02 15:51:15.790117 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:15.790086    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9488db94_ed3d_4d85_af40_5c77e36c5332.slice/crio-conmon-ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:51:15.833179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:15.833143    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"]
Nov 02 15:51:15.833437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:15.833408    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2" podUID="f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" containerName="payload-processing" containerID="cri-o://a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290" gracePeriod=30
Nov 02 15:51:15.927297 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:15.927223    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"]
Nov 02 15:51:15.927491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:15.927445    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8" podUID="f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" containerName="payload-pre-processing" containerID="cri-o://a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87" gracePeriod=30
Nov 02 15:51:16.070189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.070162    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:51:16.175101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.175069    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:51:16.199666 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.199594    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd\" (UniqueName: \"kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd\") pod \"9488db94-ed3d-4d85-af40-5c77e36c5332\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") "
Nov 02 15:51:16.199940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.199903    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca\") pod \"9488db94-ed3d-4d85-af40-5c77e36c5332\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") "
Nov 02 15:51:16.200166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.200151    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\") pod \"9488db94-ed3d-4d85-af40-5c77e36c5332\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") "
Nov 02 15:51:16.200269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.200259    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs\") pod \"9488db94-ed3d-4d85-af40-5c77e36c5332\" (UID: \"9488db94-ed3d-4d85-af40-5c77e36c5332\") "
Nov 02 15:51:16.202766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.202746    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs" pod "9488db94-ed3d-4d85-af40-5c77e36c5332" (UID: "9488db94-ed3d-4d85-af40-5c77e36c5332"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:16.204610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.204593    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd" pod "9488db94-ed3d-4d85-af40-5c77e36c5332" (UID: "9488db94-ed3d-4d85-af40-5c77e36c5332"). InnerVolumeSpecName "kube-api-access-h46sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:16.204917 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.204904    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca" pod "9488db94-ed3d-4d85-af40-5c77e36c5332" (UID: "9488db94-ed3d-4d85-af40-5c77e36c5332"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:16.207402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.206632    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls" pod "9488db94-ed3d-4d85-af40-5c77e36c5332" (UID: "9488db94-ed3d-4d85-af40-5c77e36c5332"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:16.301746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.301715    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume\") pod \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") "
Nov 02 15:51:16.301915 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.301810    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst\" (UniqueName: \"kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst\") pod \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\" (UID: \"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3\") "
Nov 02 15:51:16.302091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.302067    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume" pod "f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" (UID: "f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:16.302498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.302477    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9488db94-ed3d-4d85-af40-5c77e36c5332-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.302549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.302499    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.302549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.302509    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.302549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.302518    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9488db94-ed3d-4d85-af40-5c77e36c5332-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.302549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.302527    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46sd\" (UniqueName: \"kubernetes.io/projected/9488db94-ed3d-4d85-af40-5c77e36c5332-kube-api-access-h46sd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.304084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.304066    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst" pod "f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" (UID: "f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3"). InnerVolumeSpecName "kube-api-access-65qst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:16.329778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.329755    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:51:16.404095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.404063    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k\") pod \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") "
Nov 02 15:51:16.404255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.404239    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume\") pod \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\" (UID: \"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9\") "
Nov 02 15:51:16.404820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.404793    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qst\" (UniqueName: \"kubernetes.io/projected/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3-kube-api-access-65qst\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.405089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.405057    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume" pod "f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" (UID: "f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:16.406326 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.406300    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k" pod "f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" (UID: "f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9"). InnerVolumeSpecName "kube-api-access-2tg4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:16.450451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.450355    5041 generic.go:356] "Generic (PLEG): container finished" podID="f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" containerID="a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290" exitCode=0
Nov 02 15:51:16.450451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.450413    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2" event={"ID":"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3","Type":"ContainerDied","Data":"a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290"}
Nov 02 15:51:16.450645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.450421    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"
Nov 02 15:51:16.450645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.450481    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2" event={"ID":"f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3","Type":"ContainerDied","Data":"608f2465bfdb9d81dde5b101fa4d89455786b807499a45712ac5d2a47b18020b"}
Nov 02 15:51:16.450645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.450514    5041 scope.go:122] "RemoveContainer" containerID="a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290"
Nov 02 15:51:16.452078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.452057    5041 generic.go:356] "Generic (PLEG): container finished" podID="f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" containerID="a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87" exitCode=0
Nov 02 15:51:16.452182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.452102    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"
Nov 02 15:51:16.452182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.452086    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8" event={"ID":"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9","Type":"ContainerDied","Data":"a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87"}
Nov 02 15:51:16.452182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.452174    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8" event={"ID":"f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9","Type":"ContainerDied","Data":"83aafa07b9c1cf9afb64b411b76b0727f0e33369d12aed00b987c18dd6cc67d5"}
Nov 02 15:51:16.453968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.453947    5041 generic.go:356] "Generic (PLEG): container finished" podID="9488db94-ed3d-4d85-af40-5c77e36c5332" containerID="ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af" exitCode=0
Nov 02 15:51:16.454091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.454004    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"
Nov 02 15:51:16.454091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.454015    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" event={"ID":"9488db94-ed3d-4d85-af40-5c77e36c5332","Type":"ContainerDied","Data":"ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af"}
Nov 02 15:51:16.454091 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.454056    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" event={"ID":"9488db94-ed3d-4d85-af40-5c77e36c5332","Type":"ContainerDied","Data":"ce22ccf10273ece982e070d868242a288f27a7ae0a294cd29a28418c866bf4a9"}
Nov 02 15:51:16.464076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.464060    5041 scope.go:122] "RemoveContainer" containerID="a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290"
Nov 02 15:51:16.464398 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:16.464338    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290\": container with ID starting with a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290 not found: ID does not exist" containerID="a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290"
Nov 02 15:51:16.464471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.464396    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290"} err="failed to get container status \"a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290\": rpc error: code = NotFound desc = could not find container \"a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290\": container with ID starting with a66b35483871908c2b2d69b0887279ba89e58a97930e5922bd1558fb9af5d290 not found: ID does not exist"
Nov 02 15:51:16.464471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.464421    5041 scope.go:122] "RemoveContainer" containerID="a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87"
Nov 02 15:51:16.474610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.474592    5041 scope.go:122] "RemoveContainer" containerID="a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87"
Nov 02 15:51:16.474846 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:16.474827    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87\": container with ID starting with a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87 not found: ID does not exist" containerID="a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87"
Nov 02 15:51:16.474889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.474852    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87"} err="failed to get container status \"a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87\": rpc error: code = NotFound desc = could not find container \"a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87\": container with ID starting with a3b0119dcbd738f628ae9e11016bec777bf4d6939428a5b38911f5c8efa6cd87 not found: ID does not exist"
Nov 02 15:51:16.474889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.474868    5041 scope.go:122] "RemoveContainer" containerID="ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af"
Nov 02 15:51:16.484055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.484035    5041 scope.go:122] "RemoveContainer" containerID="ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af"
Nov 02 15:51:16.484300 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:16.484283    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af\": container with ID starting with ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af not found: ID does not exist" containerID="ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af"
Nov 02 15:51:16.484349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.484306    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af"} err="failed to get container status \"ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af\": rpc error: code = NotFound desc = could not find container \"ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af\": container with ID starting with ac4d054cb6bc327cc20bd9b600d9ea6fa6b0ddee79ed6da19fce82305de9b6af not found: ID does not exist"
Nov 02 15:51:16.506425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.506402    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.506502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.506430    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9-kube-api-access-2tg4k\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:16.511137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.511116    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"]
Nov 02 15:51:16.522727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.522700    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5721b1-75d5fcf45f-tbfv8"]
Nov 02 15:51:16.556811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.556766    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"]
Nov 02 15:51:16.561188 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.561161    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx"]
Nov 02 15:51:16.598431 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.598402    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"]
Nov 02 15:51:16.603884 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:16.603862    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5721b1-b68674996-59vw2"]
Nov 02 15:51:17.895557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:17.895524    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9488db94-ed3d-4d85-af40-5c77e36c5332" path="/var/lib/kubelet/pods/9488db94-ed3d-4d85-af40-5c77e36c5332/volumes"
Nov 02 15:51:17.895987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:17.895971    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" path="/var/lib/kubelet/pods/f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9/volumes"
Nov 02 15:51:17.896349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:17.896336    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" path="/var/lib/kubelet/pods/f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3/volumes"
Nov 02 15:51:18.414468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.414431    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main" containerID="cri-o://d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60" gracePeriod=45
Nov 02 15:51:18.701052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.701027    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:51:18.830516 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.830480    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn\" (UniqueName: \"kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn\") pod \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") "
Nov 02 15:51:18.830808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.830782    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm\") pod \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") "
Nov 02 15:51:18.830969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.830950    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home\") pod \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") "
Nov 02 15:51:18.831202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.831181    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs\") pod \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") "
Nov 02 15:51:18.831414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.831394    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir\") pod \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") "
Nov 02 15:51:18.831688 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.831666    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache\") pod \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\" (UID: \"3f5d7402-2fc1-44b7-b082-19b2ecf56b26\") "
Nov 02 15:51:18.831912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.831887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir" pod "3f5d7402-2fc1-44b7-b082-19b2ecf56b26" (UID: "3f5d7402-2fc1-44b7-b082-19b2ecf56b26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.832041 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.832019    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache" pod "3f5d7402-2fc1-44b7-b082-19b2ecf56b26" (UID: "3f5d7402-2fc1-44b7-b082-19b2ecf56b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.832152 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.832119    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home" pod "3f5d7402-2fc1-44b7-b082-19b2ecf56b26" (UID: "3f5d7402-2fc1-44b7-b082-19b2ecf56b26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.832780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.832758    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.832914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.832900    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.833014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.833000    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.837860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.837836    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm" pod "3f5d7402-2fc1-44b7-b082-19b2ecf56b26" (UID: "3f5d7402-2fc1-44b7-b082-19b2ecf56b26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.843401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.839837    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn" pod "3f5d7402-2fc1-44b7-b082-19b2ecf56b26" (UID: "3f5d7402-2fc1-44b7-b082-19b2ecf56b26"). InnerVolumeSpecName "kube-api-access-mh8tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:18.843401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.842983    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs" pod "3f5d7402-2fc1-44b7-b082-19b2ecf56b26" (UID: "3f5d7402-2fc1-44b7-b082-19b2ecf56b26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:18.934695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.934616    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8tn\" (UniqueName: \"kubernetes.io/projected/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-kube-api-access-mh8tn\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.934695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.934650    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.934695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.934666    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5d7402-2fc1-44b7-b082-19b2ecf56b26-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.974599 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:18.974562    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5721b1-8bc586475-8r5rx" podUID="9488db94-ed3d-4d85-af40-5c77e36c5332" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.50:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:51:19.497717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.497681    5041 generic.go:356] "Generic (PLEG): container finished" podID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerID="d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60" exitCode=0
Nov 02 15:51:19.497908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.497732    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"
Nov 02 15:51:19.497908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.497732    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" event={"ID":"3f5d7402-2fc1-44b7-b082-19b2ecf56b26","Type":"ContainerDied","Data":"d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60"}
Nov 02 15:51:19.497908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.497854    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq" event={"ID":"3f5d7402-2fc1-44b7-b082-19b2ecf56b26","Type":"ContainerDied","Data":"8476eb2358bb623fcbe3ba9997025c9ecd177c18e9fc4cbb820dd9ae47034bc9"}
Nov 02 15:51:19.497908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.497882    5041 scope.go:122] "RemoveContainer" containerID="d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60"
Nov 02 15:51:19.508829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.508808    5041 scope.go:122] "RemoveContainer" containerID="d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60"
Nov 02 15:51:19.509082 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:19.509059    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60\": container with ID starting with d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60 not found: ID does not exist" containerID="d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60"
Nov 02 15:51:19.509154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.509092    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60"} err="failed to get container status \"d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60\": rpc error: code = NotFound desc = could not find container \"d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60\": container with ID starting with d91b2b3d1365ae0c51620373b2d25c140b1bf9fa7db6ac7002df19d32a282b60 not found: ID does not exist"
Nov 02 15:51:19.550205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.550181    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"]
Nov 02 15:51:19.561229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.561209    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-2-w5-1cc1b2-kserve-5ccddc575-zhjqq"]
Nov 02 15:51:19.889472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:19.889401    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" path="/var/lib/kubelet/pods/3f5d7402-2fc1-44b7-b082-19b2ecf56b26/volumes"
Nov 02 15:51:22.375479 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:22.375442    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" podUID="c9c52bec-5157-4483-9df4-413a53514093" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:22.376228 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:22.375458    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:25.561337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:25.561301    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"]
Nov 02 15:51:25.561791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:25.561522    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy" containerID="cri-o://1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2" gracePeriod=30
Nov 02 15:51:29.059051 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:29.059007    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"]
Nov 02 15:51:29.090008 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:29.089976    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"]
Nov 02 15:51:30.853330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.853304    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:51:30.872873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.872842    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.872999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.872884    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.872999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.872923    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.872999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.872942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.873124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873059    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.873175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873129    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.873175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873131    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:30.873175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873163    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.873347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873224    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:30.873347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873245    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.873347 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873306    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket\") pod \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\" (UID: \"07664b84-0e99-4d03-ae3f-a3ea5518be3f\") "
Nov 02 15:51:30.873560 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873391    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:30.873625 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.873577    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:30.874058 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.874030    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.874181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.874062    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.874181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.874078    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.874181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.874092    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.874181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.874157    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:30.875450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.875402    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:30.875557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.875464    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:30.875557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.875490    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:51:30.875557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.875551    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2" pod "07664b84-0e99-4d03-ae3f-a3ea5518be3f" (UID: "07664b84-0e99-4d03-ae3f-a3ea5518be3f"). InnerVolumeSpecName "kube-api-access-28mq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:30.975294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.975254    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.975294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.975294    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.975478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.975310    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mq2\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-kube-api-access-28mq2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.975478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.975325    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:30.975478 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:30.975341    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07664b84-0e99-4d03-ae3f-a3ea5518be3f-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:31.598196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.598160    5041 generic.go:356] "Generic (PLEG): container finished" podID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerID="1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2" exitCode=0
Nov 02 15:51:31.598419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.598207    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"
Nov 02 15:51:31.598419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.598238    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" event={"ID":"07664b84-0e99-4d03-ae3f-a3ea5518be3f","Type":"ContainerDied","Data":"1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2"}
Nov 02 15:51:31.598419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.598293    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b" event={"ID":"07664b84-0e99-4d03-ae3f-a3ea5518be3f","Type":"ContainerDied","Data":"0b3adb8fe9ddb1b40996b22b34f8b7e9f78c2abd1bd719f5fc12c17dbb467160"}
Nov 02 15:51:31.598419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.598315    5041 scope.go:122] "RemoveContainer" containerID="1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2"
Nov 02 15:51:31.611020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.611000    5041 scope.go:122] "RemoveContainer" containerID="1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2"
Nov 02 15:51:31.611271 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:31.611251    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2\": container with ID starting with 1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2 not found: ID does not exist" containerID="1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2"
Nov 02 15:51:31.611320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.611279    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2"} err="failed to get container status \"1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2\": rpc error: code = NotFound desc = could not find container \"1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2\": container with ID starting with 1c9137521be6e7fca1716c2912d8c7d76e91d72cb3dcc98282037a7da5199ec2 not found: ID does not exist"
Nov 02 15:51:31.650746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.650716    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"]
Nov 02 15:51:31.661260 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.661240    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-dxr5b"]
Nov 02 15:51:31.891949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:31.891873    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" path="/var/lib/kubelet/pods/07664b84-0e99-4d03-ae3f-a3ea5518be3f/volumes"
Nov 02 15:51:32.375352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:32.375311    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" podUID="c9c52bec-5157-4483-9df4-413a53514093" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:32.375541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:32.375317    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:34.059527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.059483    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6" podUID="d880b44a-6a89-4e74-bc4c-b17f955f7cf6" containerName="payload-processing" containerID="cri-o://7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111" gracePeriod=25
Nov 02 15:51:34.090987 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.090953    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn" podUID="0832fb1c-8119-4840-afbf-9f7aac288a51" containerName="payload-pre-processing" containerID="cri-o://0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac" gracePeriod=25
Nov 02 15:51:34.275423 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:34.275387    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69736d7_8358_4118_ae72_c3517fc30b94.slice/crio-conmon-60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:51:34.276083 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:34.276056    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69736d7_8358_4118_ae72_c3517fc30b94.slice/crio-60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:51:34.400417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.394208    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:51:34.416703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.416664    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume\") pod \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") "
Nov 02 15:51:34.416891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.416776    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm\" (UniqueName: \"kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm\") pod \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\" (UID: \"d880b44a-6a89-4e74-bc4c-b17f955f7cf6\") "
Nov 02 15:51:34.417838 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.417780    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume" pod "d880b44a-6a89-4e74-bc4c-b17f955f7cf6" (UID: "d880b44a-6a89-4e74-bc4c-b17f955f7cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:34.421316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.421286    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm" pod "d880b44a-6a89-4e74-bc4c-b17f955f7cf6" (UID: "d880b44a-6a89-4e74-bc4c-b17f955f7cf6"). InnerVolumeSpecName "kube-api-access-wpfwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:34.432856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.432831    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:51:34.502420 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.502385    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"]
Nov 02 15:51:34.518120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.518084    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume\") pod \"0832fb1c-8119-4840-afbf-9f7aac288a51\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") "
Nov 02 15:51:34.518306 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.518221    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw\" (UniqueName: \"kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw\") pod \"0832fb1c-8119-4840-afbf-9f7aac288a51\" (UID: \"0832fb1c-8119-4840-afbf-9f7aac288a51\") "
Nov 02 15:51:34.519009 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.518977    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.519133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.519010    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfwm\" (UniqueName: \"kubernetes.io/projected/d880b44a-6a89-4e74-bc4c-b17f955f7cf6-kube-api-access-wpfwm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.519946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.519911    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume" pod "0832fb1c-8119-4840-afbf-9f7aac288a51" (UID: "0832fb1c-8119-4840-afbf-9f7aac288a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:34.521962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.521934    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw" pod "0832fb1c-8119-4840-afbf-9f7aac288a51" (UID: "0832fb1c-8119-4840-afbf-9f7aac288a51"). InnerVolumeSpecName "kube-api-access-kcgcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:34.534011 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.533987    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"]
Nov 02 15:51:34.555079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.555051    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:51:34.580890 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.580867    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:51:34.619444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.619417    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume\") pod \"f69736d7-8358-4118-ae72-c3517fc30b94\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") "
Nov 02 15:51:34.619610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.619490    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume\") pod \"c9c52bec-5157-4483-9df4-413a53514093\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") "
Nov 02 15:51:34.619610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.619539    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5\" (UniqueName: \"kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5\") pod \"f69736d7-8358-4118-ae72-c3517fc30b94\" (UID: \"f69736d7-8358-4118-ae72-c3517fc30b94\") "
Nov 02 15:51:34.619610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.619577    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc\" (UniqueName: \"kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc\") pod \"c9c52bec-5157-4483-9df4-413a53514093\" (UID: \"c9c52bec-5157-4483-9df4-413a53514093\") "
Nov 02 15:51:34.619954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.619930    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume" pod "f69736d7-8358-4118-ae72-c3517fc30b94" (UID: "f69736d7-8358-4118-ae72-c3517fc30b94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:34.619954 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.619947    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume" pod "c9c52bec-5157-4483-9df4-413a53514093" (UID: "c9c52bec-5157-4483-9df4-413a53514093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:34.620303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.620279    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcgcw\" (UniqueName: \"kubernetes.io/projected/0832fb1c-8119-4840-afbf-9f7aac288a51-kube-api-access-kcgcw\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.620401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.620308    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69736d7-8358-4118-ae72-c3517fc30b94-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.620401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.620322    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c52bec-5157-4483-9df4-413a53514093-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.620401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.620336    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0832fb1c-8119-4840-afbf-9f7aac288a51-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.622696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.622670    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5" pod "f69736d7-8358-4118-ae72-c3517fc30b94" (UID: "f69736d7-8358-4118-ae72-c3517fc30b94"). InnerVolumeSpecName "kube-api-access-9xbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:34.624320 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.624292    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc" pod "c9c52bec-5157-4483-9df4-413a53514093" (UID: "c9c52bec-5157-4483-9df4-413a53514093"). InnerVolumeSpecName "kube-api-access-l64bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:34.648453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.648414    5041 generic.go:356] "Generic (PLEG): container finished" podID="f69736d7-8358-4118-ae72-c3517fc30b94" containerID="60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458" exitCode=0
Nov 02 15:51:34.648612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.648459    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"
Nov 02 15:51:34.648612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.648584    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" event={"ID":"f69736d7-8358-4118-ae72-c3517fc30b94","Type":"ContainerDied","Data":"60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458"}
Nov 02 15:51:34.648763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.648626    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb" event={"ID":"f69736d7-8358-4118-ae72-c3517fc30b94","Type":"ContainerDied","Data":"5340983e3b98854b5daeecf5f7d9f746dbc87cbc54870a091e0dcb639ada337e"}
Nov 02 15:51:34.648763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.648653    5041 scope.go:122] "RemoveContainer" containerID="60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458"
Nov 02 15:51:34.652417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.652359    5041 generic.go:356] "Generic (PLEG): container finished" podID="0832fb1c-8119-4840-afbf-9f7aac288a51" containerID="0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac" exitCode=0
Nov 02 15:51:34.652626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.652601    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn" event={"ID":"0832fb1c-8119-4840-afbf-9f7aac288a51","Type":"ContainerDied","Data":"0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac"}
Nov 02 15:51:34.652712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.652657    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn" event={"ID":"0832fb1c-8119-4840-afbf-9f7aac288a51","Type":"ContainerDied","Data":"7cb8a0359afa819f165b51b277b175616791d5538f72d635424e0d783c603a81"}
Nov 02 15:51:34.652712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.652657    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"
Nov 02 15:51:34.654328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.654304    5041 generic.go:356] "Generic (PLEG): container finished" podID="c9c52bec-5157-4483-9df4-413a53514093" containerID="23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6" exitCode=0
Nov 02 15:51:34.654579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.654556    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" event={"ID":"c9c52bec-5157-4483-9df4-413a53514093","Type":"ContainerDied","Data":"23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6"}
Nov 02 15:51:34.654728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.654712    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs" event={"ID":"c9c52bec-5157-4483-9df4-413a53514093","Type":"ContainerDied","Data":"44a801a11a5d8879251c004289178b121cae767e17b1411ca15d5450c488aa9d"}
Nov 02 15:51:34.654881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.654868    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"
Nov 02 15:51:34.661438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.661413    5041 generic.go:356] "Generic (PLEG): container finished" podID="d880b44a-6a89-4e74-bc4c-b17f955f7cf6" containerID="7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111" exitCode=0
Nov 02 15:51:34.661559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.661487    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"
Nov 02 15:51:34.661559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.661510    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6" event={"ID":"d880b44a-6a89-4e74-bc4c-b17f955f7cf6","Type":"ContainerDied","Data":"7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111"}
Nov 02 15:51:34.661559 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.661547    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6" event={"ID":"d880b44a-6a89-4e74-bc4c-b17f955f7cf6","Type":"ContainerDied","Data":"fbef3c8562f3ab142d0d0efa8a950c4c734ff4854e3c2f2c98ce10caa850e1d4"}
Nov 02 15:51:34.685066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.685047    5041 scope.go:122] "RemoveContainer" containerID="60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458"
Nov 02 15:51:34.685801 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:34.685777    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458\": container with ID starting with 60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458 not found: ID does not exist" containerID="60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458"
Nov 02 15:51:34.685893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.685805    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458"} err="failed to get container status \"60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458\": rpc error: code = NotFound desc = could not find container \"60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458\": container with ID starting with 60e1bbbe48853379456728dbafd6a1943461e572beea28c5220b79e4ac4a8458 not found: ID does not exist"
Nov 02 15:51:34.685893 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.685823    5041 scope.go:122] "RemoveContainer" containerID="0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac"
Nov 02 15:51:34.698222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.698196    5041 scope.go:122] "RemoveContainer" containerID="0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac"
Nov 02 15:51:34.698515 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:34.698492    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac\": container with ID starting with 0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac not found: ID does not exist" containerID="0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac"
Nov 02 15:51:34.698602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.698524    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac"} err="failed to get container status \"0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac\": rpc error: code = NotFound desc = could not find container \"0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac\": container with ID starting with 0b8ed5ccd3154dbe945bd334fc590f1520a7028d94bbfa5fd9e43acea60203ac not found: ID does not exist"
Nov 02 15:51:34.698602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.698546    5041 scope.go:122] "RemoveContainer" containerID="23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6"
Nov 02 15:51:34.708898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.708876    5041 scope.go:122] "RemoveContainer" containerID="23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6"
Nov 02 15:51:34.709167 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:34.709147    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6\": container with ID starting with 23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6 not found: ID does not exist" containerID="23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6"
Nov 02 15:51:34.709256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.709178    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6"} err="failed to get container status \"23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6\": rpc error: code = NotFound desc = could not find container \"23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6\": container with ID starting with 23392f4a469427f75090ac96b057813fe422f8c7f4bd1ec2e3f54352c1a15aa6 not found: ID does not exist"
Nov 02 15:51:34.709256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.709199    5041 scope.go:122] "RemoveContainer" containerID="7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111"
Nov 02 15:51:34.718580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.718561    5041 scope.go:122] "RemoveContainer" containerID="7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111"
Nov 02 15:51:34.718827 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:34.718806    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111\": container with ID starting with 7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111 not found: ID does not exist" containerID="7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111"
Nov 02 15:51:34.718880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.718832    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111"} err="failed to get container status \"7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111\": rpc error: code = NotFound desc = could not find container \"7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111\": container with ID starting with 7270b7028eaa2168918a5b7cdab3a47475cb1684538018426970d1d117e08111 not found: ID does not exist"
Nov 02 15:51:34.721652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.721630    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbp5\" (UniqueName: \"kubernetes.io/projected/f69736d7-8358-4118-ae72-c3517fc30b94-kube-api-access-9xbp5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.721712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.721656    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64bc\" (UniqueName: \"kubernetes.io/projected/c9c52bec-5157-4483-9df4-413a53514093-kube-api-access-l64bc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:34.728118 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.728094    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"]
Nov 02 15:51:34.741394 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.741332    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1cc1b2-869f6cd77f-s5gkb"]
Nov 02 15:51:34.786064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.786033    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"]
Nov 02 15:51:34.790573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.790540    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1cc1b2-887d9d477-tmqrs"]
Nov 02 15:51:34.821992 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.821921    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"]
Nov 02 15:51:34.827162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.827135    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-5cf98cc995-sfrl6"]
Nov 02 15:51:34.856568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.856539    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"]
Nov 02 15:51:34.864159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:34.864134    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-84c4b46847-q59kn"]
Nov 02 15:51:35.898435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:35.898395    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0832fb1c-8119-4840-afbf-9f7aac288a51" path="/var/lib/kubelet/pods/0832fb1c-8119-4840-afbf-9f7aac288a51/volumes"
Nov 02 15:51:35.898993 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:35.898970    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c52bec-5157-4483-9df4-413a53514093" path="/var/lib/kubelet/pods/c9c52bec-5157-4483-9df4-413a53514093/volumes"
Nov 02 15:51:35.899427 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:35.899407    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d880b44a-6a89-4e74-bc4c-b17f955f7cf6" path="/var/lib/kubelet/pods/d880b44a-6a89-4e74-bc4c-b17f955f7cf6/volumes"
Nov 02 15:51:35.899860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:35.899841    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" path="/var/lib/kubelet/pods/f69736d7-8358-4118-ae72-c3517fc30b94/volumes"
Nov 02 15:51:36.769999 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:36.769963    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:51:39.502972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:39.502922    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerName="payload-processing" containerID="cri-o://fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c" gracePeriod=25
Nov 02 15:51:39.535074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:39.535024    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerName="payload-pre-processing" containerID="cri-o://ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a" gracePeriod=25
Nov 02 15:51:45.171873 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.171825    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"]
Nov 02 15:51:45.172352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.172053    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr" podUID="23b2d51e-6655-487e-8f08-0e76ac12e88c" containerName="maas-api" containerID="cri-o://d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f" gracePeriod=30
Nov 02 15:51:45.466176 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.466147    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:51:45.561491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.561388    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs\") pod \"23b2d51e-6655-487e-8f08-0e76ac12e88c\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") "
Nov 02 15:51:45.561491 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.561484    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls\") pod \"23b2d51e-6655-487e-8f08-0e76ac12e88c\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") "
Nov 02 15:51:45.561741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.561524    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca\") pod \"23b2d51e-6655-487e-8f08-0e76ac12e88c\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") "
Nov 02 15:51:45.561741 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.561600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566\" (UniqueName: \"kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566\") pod \"23b2d51e-6655-487e-8f08-0e76ac12e88c\" (UID: \"23b2d51e-6655-487e-8f08-0e76ac12e88c\") "
Nov 02 15:51:45.564477 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.563772    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca" pod "23b2d51e-6655-487e-8f08-0e76ac12e88c" (UID: "23b2d51e-6655-487e-8f08-0e76ac12e88c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:45.568630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.568520    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566" pod "23b2d51e-6655-487e-8f08-0e76ac12e88c" (UID: "23b2d51e-6655-487e-8f08-0e76ac12e88c"). InnerVolumeSpecName "kube-api-access-vz566". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:45.568630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.568595    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs" pod "23b2d51e-6655-487e-8f08-0e76ac12e88c" (UID: "23b2d51e-6655-487e-8f08-0e76ac12e88c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:45.568984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.568947    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls" pod "23b2d51e-6655-487e-8f08-0e76ac12e88c" (UID: "23b2d51e-6655-487e-8f08-0e76ac12e88c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:45.663759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.663715    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:45.663759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.663745    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b2d51e-6655-487e-8f08-0e76ac12e88c-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:45.663759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.663763    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b2d51e-6655-487e-8f08-0e76ac12e88c-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:45.664006 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.663775    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz566\" (UniqueName: \"kubernetes.io/projected/23b2d51e-6655-487e-8f08-0e76ac12e88c-kube-api-access-vz566\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:45.754799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.754716    5041 generic.go:356] "Generic (PLEG): container finished" podID="23b2d51e-6655-487e-8f08-0e76ac12e88c" containerID="d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f" exitCode=0
Nov 02 15:51:45.754799 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.754764    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"
Nov 02 15:51:45.754978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.754802    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr" event={"ID":"23b2d51e-6655-487e-8f08-0e76ac12e88c","Type":"ContainerDied","Data":"d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f"}
Nov 02 15:51:45.754978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.754858    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr" event={"ID":"23b2d51e-6655-487e-8f08-0e76ac12e88c","Type":"ContainerDied","Data":"2f899cc9603bcf81b910fc76e6f9d750cec229af16ef1588c07bfaeaf3c67b9a"}
Nov 02 15:51:45.754978 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.754883    5041 scope.go:122] "RemoveContainer" containerID="d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f"
Nov 02 15:51:45.767191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.767171    5041 scope.go:122] "RemoveContainer" containerID="d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f"
Nov 02 15:51:45.767463 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:45.767442    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f\": container with ID starting with d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f not found: ID does not exist" containerID="d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f"
Nov 02 15:51:45.767528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.767473    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f"} err="failed to get container status \"d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f\": rpc error: code = NotFound desc = could not find container \"d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f\": container with ID starting with d372d3b0f41692a75aee9d8ad74a95fd9d71061e4bceb045d7222b4e90437b8f not found: ID does not exist"
Nov 02 15:51:45.811753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.811703    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"]
Nov 02 15:51:45.815498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.815477    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8a3fb0-74d986fbcb-9x2tr"]
Nov 02 15:51:45.889610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:45.889580    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b2d51e-6655-487e-8f08-0e76ac12e88c" path="/var/lib/kubelet/pods/23b2d51e-6655-487e-8f08-0e76ac12e88c/volumes"
Nov 02 15:51:46.992811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:46.992773    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:46.992811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:46.992799    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:47.157914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.157869    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"]
Nov 02 15:51:47.158110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.158085    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" podUID="a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" containerName="maas-api" containerID="cri-o://5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d" gracePeriod=30
Nov 02 15:51:47.364892 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.364851    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"]
Nov 02 15:51:47.365132 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.365102    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c" podUID="5a6c57dc-9bc0-4045-b897-10c1a9f92145" containerName="payload-pre-processing" containerID="cri-o://b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98" gracePeriod=30
Nov 02 15:51:47.507499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.507465    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:51:47.588048 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.588009    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4\" (UniqueName: \"kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4\") pod \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") "
Nov 02 15:51:47.588252 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.588143    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca\") pod \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") "
Nov 02 15:51:47.590823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.590796    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"]
Nov 02 15:51:47.591159 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.591129    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng" podUID="ef8acb4a-4891-4ae5-a8c6-f30721f52a08" containerName="maas-api" containerID="cri-o://8763d2e978b9983277b64bf385cf999a8e8a9232d607703208e4e9dda767ef90" gracePeriod=30
Nov 02 15:51:47.594055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.594024    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\") pod \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") "
Nov 02 15:51:47.594179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.594163    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs\") pod \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\" (UID: \"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a\") "
Nov 02 15:51:47.595502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.595461    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca" pod "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" (UID: "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:47.598269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.598240    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4" pod "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" (UID: "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a"). InnerVolumeSpecName "kube-api-access-pwjp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:47.598418 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.598343    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls" pod "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" (UID: "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:47.599200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.598705    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs" pod "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" (UID: "a394f683-c6d7-42cb-82d2-5fe27e0ccb9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:47.696636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.696604    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjp4\" (UniqueName: \"kubernetes.io/projected/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-kube-api-access-pwjp4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.696636 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.696638    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.696792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.696652    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.696792 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.696665    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.706633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.706603    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:51:47.777054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.776795    5041 generic.go:356] "Generic (PLEG): container finished" podID="ef8acb4a-4891-4ae5-a8c6-f30721f52a08" containerID="8763d2e978b9983277b64bf385cf999a8e8a9232d607703208e4e9dda767ef90" exitCode=0
Nov 02 15:51:47.777458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.777322    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng" event={"ID":"ef8acb4a-4891-4ae5-a8c6-f30721f52a08","Type":"ContainerDied","Data":"8763d2e978b9983277b64bf385cf999a8e8a9232d607703208e4e9dda767ef90"}
Nov 02 15:51:47.779900 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.779873    5041 generic.go:356] "Generic (PLEG): container finished" podID="a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" containerID="5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d" exitCode=0
Nov 02 15:51:47.780223 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.780197    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" event={"ID":"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a","Type":"ContainerDied","Data":"5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d"}
Nov 02 15:51:47.780409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.780387    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t" event={"ID":"a394f683-c6d7-42cb-82d2-5fe27e0ccb9a","Type":"ContainerDied","Data":"655d788395e35330016d05cdbf96624e9d55d9650e5ffab2ffe41390ad854558"}
Nov 02 15:51:47.780566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.780549    5041 scope.go:122] "RemoveContainer" containerID="5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d"
Nov 02 15:51:47.780825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.780805    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"
Nov 02 15:51:47.784704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.784635    5041 generic.go:356] "Generic (PLEG): container finished" podID="5a6c57dc-9bc0-4045-b897-10c1a9f92145" containerID="b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98" exitCode=0
Nov 02 15:51:47.784817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.784794    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c" event={"ID":"5a6c57dc-9bc0-4045-b897-10c1a9f92145","Type":"ContainerDied","Data":"b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98"}
Nov 02 15:51:47.784913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.784837    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c" event={"ID":"5a6c57dc-9bc0-4045-b897-10c1a9f92145","Type":"ContainerDied","Data":"546ddcaf124436508d2c1553df25b60b890235c6be0c8a39ea01838e5327e5a4"}
Nov 02 15:51:47.784976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.784919    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"
Nov 02 15:51:47.795453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.795405    5041 scope.go:122] "RemoveContainer" containerID="5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d"
Nov 02 15:51:47.795781 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:47.795754    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d\": container with ID starting with 5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d not found: ID does not exist" containerID="5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d"
Nov 02 15:51:47.795871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.795788    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d"} err="failed to get container status \"5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d\": rpc error: code = NotFound desc = could not find container \"5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d\": container with ID starting with 5feef50c481fdae99679017c267b0150d9558d477a563dfc70c45fe268ccc23d not found: ID does not exist"
Nov 02 15:51:47.795871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.795812    5041 scope.go:122] "RemoveContainer" containerID="b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98"
Nov 02 15:51:47.797918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.797894    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume\") pod \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") "
Nov 02 15:51:47.798024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.797965    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9\" (UniqueName: \"kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9\") pod \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\" (UID: \"5a6c57dc-9bc0-4045-b897-10c1a9f92145\") "
Nov 02 15:51:47.799820 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.799459    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume" pod "5a6c57dc-9bc0-4045-b897-10c1a9f92145" (UID: "5a6c57dc-9bc0-4045-b897-10c1a9f92145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:47.802819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.802794    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9" pod "5a6c57dc-9bc0-4045-b897-10c1a9f92145" (UID: "5a6c57dc-9bc0-4045-b897-10c1a9f92145"). InnerVolumeSpecName "kube-api-access-85hg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:47.813575 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.813551    5041 scope.go:122] "RemoveContainer" containerID="b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98"
Nov 02 15:51:47.814021 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:47.813994    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98\": container with ID starting with b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98 not found: ID does not exist" containerID="b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98"
Nov 02 15:51:47.814148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.814027    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98"} err="failed to get container status \"b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98\": rpc error: code = NotFound desc = could not find container \"b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98\": container with ID starting with b4ae91b6ef71b2e91fa107a3f8b4a1d3b38b05237014989f3be3fb23d4a39c98 not found: ID does not exist"
Nov 02 15:51:47.850074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.850040    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"]
Nov 02 15:51:47.853748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.853721    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d79991-7c5949fdcc-s5r9t"]
Nov 02 15:51:47.877192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.877172    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:51:47.890133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.890110    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" path="/var/lib/kubelet/pods/a394f683-c6d7-42cb-82d2-5fe27e0ccb9a/volumes"
Nov 02 15:51:47.900144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.900122    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c57dc-9bc0-4045-b897-10c1a9f92145-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.900232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:47.900146    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hg9\" (UniqueName: \"kubernetes.io/projected/5a6c57dc-9bc0-4045-b897-10c1a9f92145-kube-api-access-85hg9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:48.000676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.000645    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs\") pod \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") "
Nov 02 15:51:48.001116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.000759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca\") pod \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") "
Nov 02 15:51:48.001116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.000881    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2\" (UniqueName: \"kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2\") pod \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") "
Nov 02 15:51:48.001116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.001053    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls\") pod \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\" (UID: \"ef8acb4a-4891-4ae5-a8c6-f30721f52a08\") "
Nov 02 15:51:48.001116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.001056    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca" pod "ef8acb4a-4891-4ae5-a8c6-f30721f52a08" (UID: "ef8acb4a-4891-4ae5-a8c6-f30721f52a08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:48.001589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.001564    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:48.002819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.002799    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs" pod "ef8acb4a-4891-4ae5-a8c6-f30721f52a08" (UID: "ef8acb4a-4891-4ae5-a8c6-f30721f52a08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:48.002942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.002921    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls" pod "ef8acb4a-4891-4ae5-a8c6-f30721f52a08" (UID: "ef8acb4a-4891-4ae5-a8c6-f30721f52a08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:48.003004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.002985    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2" pod "ef8acb4a-4891-4ae5-a8c6-f30721f52a08" (UID: "ef8acb4a-4891-4ae5-a8c6-f30721f52a08"). InnerVolumeSpecName "kube-api-access-d99j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:48.102938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.102857    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:48.102938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.102891    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:48.102938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.102902    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99j2\" (UniqueName: \"kubernetes.io/projected/ef8acb4a-4891-4ae5-a8c6-f30721f52a08-kube-api-access-d99j2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:48.130346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.130289    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"]
Nov 02 15:51:48.136081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.136048    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7966c5859-xsj6c"]
Nov 02 15:51:48.794637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.794605    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"
Nov 02 15:51:48.794637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.794629    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng" event={"ID":"ef8acb4a-4891-4ae5-a8c6-f30721f52a08","Type":"ContainerDied","Data":"4e7546ff5d1d5e573168f27e17a83c6f424b01d0ea4751e16a397fb07e750dfb"}
Nov 02 15:51:48.794903 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.794695    5041 scope.go:122] "RemoveContainer" containerID="8763d2e978b9983277b64bf385cf999a8e8a9232d607703208e4e9dda767ef90"
Nov 02 15:51:48.847633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.847602    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"]
Nov 02 15:51:48.858809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:48.858781    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-1cc1b2-dfcd7478d-jjcng"]
Nov 02 15:51:49.890527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:49.890490    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6c57dc-9bc0-4045-b897-10c1a9f92145" path="/var/lib/kubelet/pods/5a6c57dc-9bc0-4045-b897-10c1a9f92145/volumes"
Nov 02 15:51:49.891201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:49.891177    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8acb4a-4891-4ae5-a8c6-f30721f52a08" path="/var/lib/kubelet/pods/ef8acb4a-4891-4ae5-a8c6-f30721f52a08/volumes"
Nov 02 15:51:53.143667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:53.143630    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"]
Nov 02 15:51:53.144155 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:53.143879    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy" containerID="cri-o://67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb" gracePeriod=30
Nov 02 15:51:56.825572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:56.825517    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"]
Nov 02 15:51:56.826183 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:56.825803    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy" containerID="cri-o://4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a" gracePeriod=30
Nov 02 15:51:56.993222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:56.993177    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:56.993222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:56.993202    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:51:58.420300 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.420275    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:51:58.533588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533553    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.533588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.533847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533657    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.533847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533696    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.533847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533784    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.534037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533873    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.534037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533921    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.534037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.533968    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.534037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534032    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data\") pod \"46272a17-1a46-4999-95d2-b25e6e57ddc0\" (UID: \"46272a17-1a46-4999-95d2-b25e6e57ddc0\") "
Nov 02 15:51:58.534249 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534042    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:58.534633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534397    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:58.534633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534558    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.534633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534577    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.534633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534593    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:58.534633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534619    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:58.535054 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.534755    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:58.537684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.537658    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:51:58.537840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.537721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "kube-api-access-sqjxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:58.537840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.537741    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:58.538950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.538918    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy" pod "46272a17-1a46-4999-95d2-b25e6e57ddc0" (UID: "46272a17-1a46-4999-95d2-b25e6e57ddc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635859    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635900    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjxr\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-kube-api-access-sqjxr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635916    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46272a17-1a46-4999-95d2-b25e6e57ddc0-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635930    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635943    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635958    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46272a17-1a46-4999-95d2-b25e6e57ddc0-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.636104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.635972    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46272a17-1a46-4999-95d2-b25e6e57ddc0-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:58.891208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.891088    5041 generic.go:356] "Generic (PLEG): container finished" podID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerID="67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb" exitCode=0
Nov 02 15:51:58.891208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.891150    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"
Nov 02 15:51:58.891208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.891169    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" event={"ID":"46272a17-1a46-4999-95d2-b25e6e57ddc0","Type":"ContainerDied","Data":"67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb"}
Nov 02 15:51:58.891524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.891236    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j" event={"ID":"46272a17-1a46-4999-95d2-b25e6e57ddc0","Type":"ContainerDied","Data":"9d3cef5b0dbc77d558e5fb3fd7758aa3e49213171d12d4fbb5785e8814e709e6"}
Nov 02 15:51:58.891524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.891266    5041 scope.go:122] "RemoveContainer" containerID="67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb"
Nov 02 15:51:58.903654 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.903634    5041 scope.go:122] "RemoveContainer" containerID="57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"
Nov 02 15:51:58.913578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.913558    5041 scope.go:122] "RemoveContainer" containerID="67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb"
Nov 02 15:51:58.913857 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:58.913833    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb\": container with ID starting with 67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb not found: ID does not exist" containerID="67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb"
Nov 02 15:51:58.913955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.913860    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb"} err="failed to get container status \"67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb\": rpc error: code = NotFound desc = could not find container \"67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb\": container with ID starting with 67d8b99da0f9f7de43b8b574d328263cd0ac3fd26ce3f1617016ed67e8d859fb not found: ID does not exist"
Nov 02 15:51:58.913955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.913876    5041 scope.go:122] "RemoveContainer" containerID="57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"
Nov 02 15:51:58.914141 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:58.914123    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080\": container with ID starting with 57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080 not found: ID does not exist" containerID="57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"
Nov 02 15:51:58.914193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.914147    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080"} err="failed to get container status \"57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080\": rpc error: code = NotFound desc = could not find container \"57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080\": container with ID starting with 57ce62d5696a2a415e8eb97bbd390931c5fb2f3282a34fbd5f504cd12e9b7080 not found: ID does not exist"
Nov 02 15:51:58.940788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.940765    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"]
Nov 02 15:51:58.941454 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941439    5041 cpu_manager.go:414] "containerMap: removing container" podUID="58285986-2b9a-4804-9494-5c61afe725d7" containerName="maas-api"
Nov 02 15:51:58.941511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941455    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58285986-2b9a-4804-9494-5c61afe725d7" containerName="maas-api"
Nov 02 15:51:58.941511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941476    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c9c52bec-5157-4483-9df4-413a53514093" containerName="payload-processing"
Nov 02 15:51:58.941511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941481    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c52bec-5157-4483-9df4-413a53514093" containerName="payload-processing"
Nov 02 15:51:58.941511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941499    5041 cpu_manager.go:414] "containerMap: removing container" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy"
Nov 02 15:51:58.941511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941505    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941515    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941519    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941528    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941533    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941541    5041 cpu_manager.go:414] "containerMap: removing container" podUID="23b2d51e-6655-487e-8f08-0e76ac12e88c" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941545    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b2d51e-6655-487e-8f08-0e76ac12e88c" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941554    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941559    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941567    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5a6c57dc-9bc0-4045-b897-10c1a9f92145" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941571    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6c57dc-9bc0-4045-b897-10c1a9f92145" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941586    5041 cpu_manager.go:414] "containerMap: removing container" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941591    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941597    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941601    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941609    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" containerName="payload-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941613    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" containerName="payload-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941622    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ef8acb4a-4891-4ae5-a8c6-f30721f52a08" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941627    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8acb4a-4891-4ae5-a8c6-f30721f52a08" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941635    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0832fb1c-8119-4840-afbf-9f7aac288a51" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941640    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0832fb1c-8119-4840-afbf-9f7aac288a51" containerName="payload-pre-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941648    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9488db94-ed3d-4d85-af40-5c77e36c5332" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941654    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9488db94-ed3d-4d85-af40-5c77e36c5332" containerName="maas-api"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941660    5041 cpu_manager.go:414] "containerMap: removing container" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941665    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941675    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d880b44a-6a89-4e74-bc4c-b17f955f7cf6" containerName="payload-processing"
Nov 02 15:51:58.941677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941680    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d880b44a-6a89-4e74-bc4c-b17f955f7cf6" containerName="payload-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941798    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="58285986-2b9a-4804-9494-5c61afe725d7" containerName="maas-api"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941809    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c52bec-5157-4483-9df4-413a53514093" containerName="payload-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941818    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941824    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a394f683-c6d7-42cb-82d2-5fe27e0ccb9a" containerName="maas-api"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941830    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5d7402-2fc1-44b7-b082-19b2ecf56b26" containerName="main"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941835    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b2d51e-6655-487e-8f08-0e76ac12e88c" containerName="maas-api"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941844    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69736d7-8358-4118-ae72-c3517fc30b94" containerName="payload-pre-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941856    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6c57dc-9bc0-4045-b897-10c1a9f92145" containerName="payload-pre-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941863    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e0f6d5-33f9-43ee-bc6a-7c678a6a39c9" containerName="payload-pre-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941870    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9de6c10-6c45-4b14-a1ed-ebc45ef8e3f3" containerName="payload-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941877    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8acb4a-4891-4ae5-a8c6-f30721f52a08" containerName="maas-api"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941882    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0832fb1c-8119-4840-afbf-9f7aac288a51" containerName="payload-pre-processing"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941888    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9488db94-ed3d-4d85-af40-5c77e36c5332" containerName="maas-api"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941893    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="07664b84-0e99-4d03-ae3f-a3ea5518be3f" containerName="istio-proxy"
Nov 02 15:51:58.942544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.941900    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d880b44a-6a89-4e74-bc4c-b17f955f7cf6" containerName="payload-processing"
Nov 02 15:51:58.946757 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.946740    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:58.947576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.947558    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"]
Nov 02 15:51:58.972752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.963260    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8a3fb0-openshift-default-779d77bf9-q7w6j"]
Nov 02 15:51:58.972752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.968770    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a433eb-dockercfg-q85m4\""
Nov 02 15:51:58.972752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.969265    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a433eb\""
Nov 02 15:51:58.977643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:58.977617    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"]
Nov 02 15:51:59.040711 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.040683    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvbz\" (UniqueName: \"kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.040895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.040872    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.098001 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.097968    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"]
Nov 02 15:51:59.098901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.098879    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" containerName="istio-proxy"
Nov 02 15:51:59.102778 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.102758    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.124407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.124381    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"]
Nov 02 15:51:59.141887 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.141822    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcr6x\" (UniqueName: \"kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x\") pod \"payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.141999 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.141890    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvbz\" (UniqueName: \"kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.142072 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.142028    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.142137 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.142069    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.142745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.142726    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.164946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.164922    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvbz\" (UniqueName: \"kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz\") pod \"payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.242935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.242905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.243061 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.242993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcr6x\" (UniqueName: \"kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x\") pod \"payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.243739 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.243712    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.266429 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.266394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcr6x\" (UniqueName: \"kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x\") pod \"payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.275066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.275043    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.418059 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.417988    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.590841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.590814    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"]
Nov 02 15:51:59.618776 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:51:59.618748    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27bbce5_5794_4efe_9248_0716a561918f.slice/crio-54f58f38d8c13aff9d8684eda7e471591c779067804898533f1efc969acdf078 WatchSource:0}: Error finding container 54f58f38d8c13aff9d8684eda7e471591c779067804898533f1efc969acdf078: Status 404 returned error can't find the container with id 54f58f38d8c13aff9d8684eda7e471591c779067804898533f1efc969acdf078
Nov 02 15:51:59.727655 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.727610    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"]
Nov 02 15:51:59.805121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.805094    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:51:59.849358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.849328    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2\" (UniqueName: \"kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2\") pod \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") "
Nov 02 15:51:59.849509 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.849394    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume\") pod \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\" (UID: \"5704f35f-3ee9-4a3a-96b9-9d659d474ee7\") "
Nov 02 15:51:59.850284 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.850192    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume" pod "5704f35f-3ee9-4a3a-96b9-9d659d474ee7" (UID: "5704f35f-3ee9-4a3a-96b9-9d659d474ee7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:59.852724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.852680    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2" pod "5704f35f-3ee9-4a3a-96b9-9d659d474ee7" (UID: "5704f35f-3ee9-4a3a-96b9-9d659d474ee7"). InnerVolumeSpecName "kube-api-access-xjbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:59.878982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.878949    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:51:59.892074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.892046    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46272a17-1a46-4999-95d2-b25e6e57ddc0" path="/var/lib/kubelet/pods/46272a17-1a46-4999-95d2-b25e6e57ddc0/volumes"
Nov 02 15:51:59.902224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.902196    5041 generic.go:356] "Generic (PLEG): container finished" podID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerID="fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c" exitCode=0
Nov 02 15:51:59.902323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.902250    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"
Nov 02 15:51:59.902323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.902280    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" event={"ID":"5704f35f-3ee9-4a3a-96b9-9d659d474ee7","Type":"ContainerDied","Data":"fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c"}
Nov 02 15:51:59.902469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.902334    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q" event={"ID":"5704f35f-3ee9-4a3a-96b9-9d659d474ee7","Type":"ContainerDied","Data":"919e0f06c9c4b7edff268b86c8285fb018156597ad4883344ecfbf1315418b64"}
Nov 02 15:51:59.902469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.902378    5041 scope.go:122] "RemoveContainer" containerID="fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c"
Nov 02 15:51:59.905027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.904900    5041 generic.go:356] "Generic (PLEG): container finished" podID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerID="ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a" exitCode=0
Nov 02 15:51:59.905027 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.904965    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"
Nov 02 15:51:59.905157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.905030    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" event={"ID":"5c5b4e77-a767-424d-9626-b2e877f1ae7b","Type":"ContainerDied","Data":"ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a"}
Nov 02 15:51:59.905157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.905076    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc" event={"ID":"5c5b4e77-a767-424d-9626-b2e877f1ae7b","Type":"ContainerDied","Data":"e76292ef6c3ec8ddb38509f7ec61f0af722d4a8d575374e71c851120167c9f63"}
Nov 02 15:51:59.907511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.907489    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" event={"ID":"8988bd7c-4215-4f96-817f-439871ea3f75","Type":"ContainerStarted","Data":"32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69"}
Nov 02 15:51:59.907622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.907525    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" event={"ID":"8988bd7c-4215-4f96-817f-439871ea3f75","Type":"ContainerStarted","Data":"47e95d686adcf30ccb992b76efe8f1dc1fc966a97ba826efda791710dd98ba73"}
Nov 02 15:51:59.907622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.907568    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:51:59.910904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.910846    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" event={"ID":"d27bbce5-5794-4efe-9248-0716a561918f","Type":"ContainerStarted","Data":"d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330"}
Nov 02 15:51:59.910904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.910889    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" event={"ID":"d27bbce5-5794-4efe-9248-0716a561918f","Type":"ContainerStarted","Data":"54f58f38d8c13aff9d8684eda7e471591c779067804898533f1efc969acdf078"}
Nov 02 15:51:59.911079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.910922    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:51:59.917983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.917957    5041 scope.go:122] "RemoveContainer" containerID="fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c"
Nov 02 15:51:59.918240 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:59.918214    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c\": container with ID starting with fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c not found: ID does not exist" containerID="fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c"
Nov 02 15:51:59.918331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.918244    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c"} err="failed to get container status \"fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c\": rpc error: code = NotFound desc = could not find container \"fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c\": container with ID starting with fd47e37eb3a5eb48ced076b8ff15df5f5c3b2b3fdd015de21733e5ddc1dee45c not found: ID does not exist"
Nov 02 15:51:59.918331 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.918260    5041 scope.go:122] "RemoveContainer" containerID="ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a"
Nov 02 15:51:59.934402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.934380    5041 scope.go:122] "RemoveContainer" containerID="ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a"
Nov 02 15:51:59.934681 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:51:59.934659    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a\": container with ID starting with ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a not found: ID does not exist" containerID="ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a"
Nov 02 15:51:59.934755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.934685    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a"} err="failed to get container status \"ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a\": rpc error: code = NotFound desc = could not find container \"ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a\": container with ID starting with ff46f869e8c19f7dfed86fa20648137f8b61a550692f028aa5f67348b405a00a not found: ID does not exist"
Nov 02 15:51:59.951386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.951349    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume\") pod \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") "
Nov 02 15:51:59.951500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.951419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6\" (UniqueName: \"kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6\") pod \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\" (UID: \"5c5b4e77-a767-424d-9626-b2e877f1ae7b\") "
Nov 02 15:51:59.951730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.951706    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume" pod "5c5b4e77-a767-424d-9626-b2e877f1ae7b" (UID: "5c5b4e77-a767-424d-9626-b2e877f1ae7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:59.952215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.952142    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbv2\" (UniqueName: \"kubernetes.io/projected/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-kube-api-access-xjbv2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:59.952215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.952171    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5704f35f-3ee9-4a3a-96b9-9d659d474ee7-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:59.952215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.952188    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5b4e77-a767-424d-9626-b2e877f1ae7b-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:51:59.954669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.954642    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6" pod "5c5b4e77-a767-424d-9626-b2e877f1ae7b" (UID: "5c5b4e77-a767-424d-9626-b2e877f1ae7b"). InnerVolumeSpecName "kube-api-access-l58j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:59.960343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.960139    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"]
Nov 02 15:51:59.966828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:51:59.966804    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8a3fb0-579df9d9d5-dc45q"]
Nov 02 15:52:00.010979 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:00.010926    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" podStartSLOduration=2.010909871 podStartE2EDuration="2.010909871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:00.007977875 +0000 UTC m=+3372.266047324" watchObservedRunningTime="2026-11-02 15:52:00.010909871 +0000 UTC m=+3372.268979323"
Nov 02 15:52:00.051417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:00.051346    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" podStartSLOduration=1.051330772 podStartE2EDuration="1.051330772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:00.048209138 +0000 UTC m=+3372.306278588" watchObservedRunningTime="2026-11-02 15:52:00.051330772 +0000 UTC m=+3372.309400223"
Nov 02 15:52:00.053098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:00.053073    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58j6\" (UniqueName: \"kubernetes.io/projected/5c5b4e77-a767-424d-9626-b2e877f1ae7b-kube-api-access-l58j6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:00.253116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:00.253082    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"]
Nov 02 15:52:00.259129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:00.259101    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8a3fb0-696fc77c99-6jqgc"]
Nov 02 15:52:01.889555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:01.889526    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" path="/var/lib/kubelet/pods/5704f35f-3ee9-4a3a-96b9-9d659d474ee7/volumes"
Nov 02 15:52:01.889958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:01.889940    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" path="/var/lib/kubelet/pods/5c5b4e77-a767-424d-9626-b2e877f1ae7b/volumes"
Nov 02 15:52:02.113518 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.113452    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:52:02.175655 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.175622    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.175806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.175697    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.175806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.175757    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.175940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.175824    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.175940 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.175872    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.176068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.175942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.176068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176017    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.176068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176052    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.176245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176176    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket\") pod \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\" (UID: \"cf5eb9e3-d415-402a-b228-71fc41dd80fe\") "
Nov 02 15:52:02.176806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176774    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176923    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.176989    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.177253    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.177463    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.177484    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.177498    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.177510    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.177785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.177524    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.178310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.178286    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "kube-api-access-cg6v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:02.178385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.178330    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:02.178753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.178727    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:02.179874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.179850    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy" pod "cf5eb9e3-d415-402a-b228-71fc41dd80fe" (UID: "cf5eb9e3-d415-402a-b228-71fc41dd80fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:02.279049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.279019    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.279049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.279045    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6v5\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-kube-api-access-cg6v5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.279049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.279054    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.279247 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.279064    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5eb9e3-d415-402a-b228-71fc41dd80fe-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:02.938494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.938456    5041 generic.go:356] "Generic (PLEG): container finished" podID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerID="4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a" exitCode=0
Nov 02 15:52:02.938981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.938504    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"
Nov 02 15:52:02.938981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.938548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" event={"ID":"cf5eb9e3-d415-402a-b228-71fc41dd80fe","Type":"ContainerDied","Data":"4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a"}
Nov 02 15:52:02.938981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.938618    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd" event={"ID":"cf5eb9e3-d415-402a-b228-71fc41dd80fe","Type":"ContainerDied","Data":"fbd71136ccb93e5c691c9c14594a625efac21dfcb777193e6b3a7e3ab75bf88b"}
Nov 02 15:52:02.938981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.938655    5041 scope.go:122] "RemoveContainer" containerID="4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a"
Nov 02 15:52:02.950610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.950420    5041 scope.go:122] "RemoveContainer" containerID="4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a"
Nov 02 15:52:02.950751 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:02.950732    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a\": container with ID starting with 4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a not found: ID does not exist" containerID="4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a"
Nov 02 15:52:02.950800 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:02.950758    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a"} err="failed to get container status \"4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a\": rpc error: code = NotFound desc = could not find container \"4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a\": container with ID starting with 4014eb238668662ee5386331e91f1fe73b49d5997eb048e0b9825765ea48e51a not found: ID does not exist"
Nov 02 15:52:03.002927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:03.002899    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"]
Nov 02 15:52:03.014298 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:03.014271    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-vzhmd"]
Nov 02 15:52:03.888775 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:03.888743    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" path="/var/lib/kubelet/pods/cf5eb9e3-d415-402a-b228-71fc41dd80fe/volumes"
Nov 02 15:52:04.161558 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:04.161481    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"]
Nov 02 15:52:04.194837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:04.194809    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"]
Nov 02 15:52:06.769039 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:06.769001    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:52:09.162115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:09.162071    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerName="payload-processing" containerID="cri-o://679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009" gracePeriod=25
Nov 02 15:52:09.195421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:09.195387    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerName="payload-pre-processing" containerID="cri-o://9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba" gracePeriod=25
Nov 02 15:52:10.924440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:10.924404    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:52:10.924859 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:10.924490    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:52:13.182834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:13.182799    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:52:13.184649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:13.184622    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:52:21.019948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:21.019919    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp_c22dfaed-e76a-4b46-b522-9f4a6d119018/payload-processing/0.log"
Nov 02 15:52:21.206648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:21.206622    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp_c22dfaed-e76a-4b46-b522-9f4a6d119018/payload-processing/0.log"
Nov 02 15:52:22.306886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:22.306851    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"]
Nov 02 15:52:22.307290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:22.307057    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy" containerID="cri-o://8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3" gracePeriod=30
Nov 02 15:52:22.581123 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:22.581036    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"]
Nov 02 15:52:22.581832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:22.581298    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc" podUID="6e0714ad-acf9-4929-a875-81c3b757d8b4" containerName="maas-api" containerID="cri-o://4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88" gracePeriod=30
Nov 02 15:52:22.862395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:22.862329    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:52:23.002824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.002787    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls\") pod \"6e0714ad-acf9-4929-a875-81c3b757d8b4\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") "
Nov 02 15:52:23.003034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.002922    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs\") pod \"6e0714ad-acf9-4929-a875-81c3b757d8b4\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") "
Nov 02 15:52:23.003034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.002964    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4\" (UniqueName: \"kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4\") pod \"6e0714ad-acf9-4929-a875-81c3b757d8b4\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") "
Nov 02 15:52:23.003034 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.003023    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca\") pod \"6e0714ad-acf9-4929-a875-81c3b757d8b4\" (UID: \"6e0714ad-acf9-4929-a875-81c3b757d8b4\") "
Nov 02 15:52:23.003845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.003492    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca" pod "6e0714ad-acf9-4929-a875-81c3b757d8b4" (UID: "6e0714ad-acf9-4929-a875-81c3b757d8b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:23.003845 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.003625    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e0714ad-acf9-4929-a875-81c3b757d8b4-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:23.005277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.005250    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs" pod "6e0714ad-acf9-4929-a875-81c3b757d8b4" (UID: "6e0714ad-acf9-4929-a875-81c3b757d8b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:23.005439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.005274    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls" pod "6e0714ad-acf9-4929-a875-81c3b757d8b4" (UID: "6e0714ad-acf9-4929-a875-81c3b757d8b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:23.005596 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.005575    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4" pod "6e0714ad-acf9-4929-a875-81c3b757d8b4" (UID: "6e0714ad-acf9-4929-a875-81c3b757d8b4"). InnerVolumeSpecName "kube-api-access-frhn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:23.100353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.100266    5041 generic.go:356] "Generic (PLEG): container finished" podID="6e0714ad-acf9-4929-a875-81c3b757d8b4" containerID="4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88" exitCode=0
Nov 02 15:52:23.100353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.100311    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"
Nov 02 15:52:23.100617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.100347    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc" event={"ID":"6e0714ad-acf9-4929-a875-81c3b757d8b4","Type":"ContainerDied","Data":"4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88"}
Nov 02 15:52:23.100617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.100421    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc" event={"ID":"6e0714ad-acf9-4929-a875-81c3b757d8b4","Type":"ContainerDied","Data":"adaa8db17ae1361a5df654e4b8e7dbfa5b37637bdd54d6c8f44489ac52be99c4"}
Nov 02 15:52:23.100617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.100453    5041 scope.go:122] "RemoveContainer" containerID="4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88"
Nov 02 15:52:23.104704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.104679    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:23.104704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.104703    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhn4\" (UniqueName: \"kubernetes.io/projected/6e0714ad-acf9-4929-a875-81c3b757d8b4-kube-api-access-frhn4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:23.104850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.104713    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e0714ad-acf9-4929-a875-81c3b757d8b4-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:23.112163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.112140    5041 scope.go:122] "RemoveContainer" containerID="4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88"
Nov 02 15:52:23.112499 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:23.112476    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88\": container with ID starting with 4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88 not found: ID does not exist" containerID="4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88"
Nov 02 15:52:23.112606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.112508    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88"} err="failed to get container status \"4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88\": rpc error: code = NotFound desc = could not find container \"4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88\": container with ID starting with 4eb61315f2f6bc3faefa5083a9ecf91c8969b96eaedb29edd2aa2034cbd72d88 not found: ID does not exist"
Nov 02 15:52:23.153015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.152988    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"]
Nov 02 15:52:23.158724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.158703    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-453864-595b59fbc5-gcdlc"]
Nov 02 15:52:23.182819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.182792    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:52:23.184747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.184714    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:52:23.889106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:23.889067    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0714ad-acf9-4929-a875-81c3b757d8b4" path="/var/lib/kubelet/pods/6e0714ad-acf9-4929-a875-81c3b757d8b4/volumes"
Nov 02 15:52:27.595867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.595840    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:52:27.758566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758533    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.758566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758572    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.758809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758639    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.758809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758657    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.758809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758735    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.758809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758767    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.759044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758842    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.759044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758898    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.759044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758937    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:27.759044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.758981    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token\") pod \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\" (UID: \"c58e3d37-fc86-46dd-8cb4-fab66bca973d\") "
Nov 02 15:52:27.759268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759056    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:27.759268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759037    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:27.759268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759115    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:27.759268 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759150    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:27.759525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759396    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.759525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759419    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.759525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759427    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.759525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759436    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.759525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.759445    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.761104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.761073    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "kube-api-access-c6nxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:27.761215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.761111    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:27.761325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.761305    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:27.761399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.761338    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo" pod "c58e3d37-fc86-46dd-8cb4-fab66bca973d" (UID: "c58e3d37-fc86-46dd-8cb4-fab66bca973d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:27.860939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.860883    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.860939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.860907    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.860939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.860917    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e3d37-fc86-46dd-8cb4-fab66bca973d-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:27.860939 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:27.860926    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nxz\" (UniqueName: \"kubernetes.io/projected/c58e3d37-fc86-46dd-8cb4-fab66bca973d-kube-api-access-c6nxz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:28.139969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.139900    5041 generic.go:356] "Generic (PLEG): container finished" podID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerID="8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3" exitCode=0
Nov 02 15:52:28.139969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.139941    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"
Nov 02 15:52:28.140129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.139978    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" event={"ID":"c58e3d37-fc86-46dd-8cb4-fab66bca973d","Type":"ContainerDied","Data":"8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3"}
Nov 02 15:52:28.140129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.140031    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx" event={"ID":"c58e3d37-fc86-46dd-8cb4-fab66bca973d","Type":"ContainerDied","Data":"0e2b407b4117638df2eab1e3dfeb904b8670743230417cc3330b8f77b6fc57e7"}
Nov 02 15:52:28.140129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.140052    5041 scope.go:122] "RemoveContainer" containerID="8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3"
Nov 02 15:52:28.150717 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.150691    5041 scope.go:122] "RemoveContainer" containerID="8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3"
Nov 02 15:52:28.150987 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:28.150969    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3\": container with ID starting with 8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3 not found: ID does not exist" containerID="8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3"
Nov 02 15:52:28.151063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.150993    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3"} err="failed to get container status \"8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3\": rpc error: code = NotFound desc = could not find container \"8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3\": container with ID starting with 8f05d589c3d6c1ae40f5f9ee862683c68316357262daed5b25e0c466e4e701f3 not found: ID does not exist"
Nov 02 15:52:28.192674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.192645    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"]
Nov 02 15:52:28.196895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:28.196874    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-gs4cx"]
Nov 02 15:52:29.464867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.464838    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:52:29.529170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.529140    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:52:29.579604 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.579574    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume\") pod \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") "
Nov 02 15:52:29.579752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.579642    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh\" (UniqueName: \"kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh\") pod \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\" (UID: \"66419e73-7d9e-45f8-bf42-d38f673f4d7b\") "
Nov 02 15:52:29.579752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.579715    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd\" (UniqueName: \"kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd\") pod \"e617804b-9177-4740-b61e-9ae08e8f45a0\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") "
Nov 02 15:52:29.579942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.579918    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume" pod "66419e73-7d9e-45f8-bf42-d38f673f4d7b" (UID: "66419e73-7d9e-45f8-bf42-d38f673f4d7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:29.580204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.580186    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66419e73-7d9e-45f8-bf42-d38f673f4d7b-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:29.581849 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.581823    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd" pod "e617804b-9177-4740-b61e-9ae08e8f45a0" (UID: "e617804b-9177-4740-b61e-9ae08e8f45a0"). InnerVolumeSpecName "kube-api-access-hb4pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:29.582200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.582175    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh" pod "66419e73-7d9e-45f8-bf42-d38f673f4d7b" (UID: "66419e73-7d9e-45f8-bf42-d38f673f4d7b"). InnerVolumeSpecName "kube-api-access-k77lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:29.680829 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.680764    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume\") pod \"e617804b-9177-4740-b61e-9ae08e8f45a0\" (UID: \"e617804b-9177-4740-b61e-9ae08e8f45a0\") "
Nov 02 15:52:29.681007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.680991    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77lh\" (UniqueName: \"kubernetes.io/projected/66419e73-7d9e-45f8-bf42-d38f673f4d7b-kube-api-access-k77lh\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:29.681082 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.681007    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4pd\" (UniqueName: \"kubernetes.io/projected/e617804b-9177-4740-b61e-9ae08e8f45a0-kube-api-access-hb4pd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:29.681163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.681140    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume" pod "e617804b-9177-4740-b61e-9ae08e8f45a0" (UID: "e617804b-9177-4740-b61e-9ae08e8f45a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:29.781934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.781903    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617804b-9177-4740-b61e-9ae08e8f45a0-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:29.889520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:29.889495    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" path="/var/lib/kubelet/pods/c58e3d37-fc86-46dd-8cb4-fab66bca973d/volumes"
Nov 02 15:52:30.164977 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.164886    5041 generic.go:356] "Generic (PLEG): container finished" podID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerID="9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba" exitCode=0
Nov 02 15:52:30.165151 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.165069    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" event={"ID":"e617804b-9177-4740-b61e-9ae08e8f45a0","Type":"ContainerDied","Data":"9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba"}
Nov 02 15:52:30.165214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.165161    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn" event={"ID":"e617804b-9177-4740-b61e-9ae08e8f45a0","Type":"ContainerDied","Data":"dda3d0262fc8a81926ffd9aa4bbcae774ac5a9a5eab2714301d1b8f44a4a7f52"}
Nov 02 15:52:30.165214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.165200    5041 scope.go:122] "RemoveContainer" containerID="9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba"
Nov 02 15:52:30.165444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.165423    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"
Nov 02 15:52:30.168287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.168259    5041 generic.go:356] "Generic (PLEG): container finished" podID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerID="679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009" exitCode=0
Nov 02 15:52:30.168541 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.168514    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" event={"ID":"66419e73-7d9e-45f8-bf42-d38f673f4d7b","Type":"ContainerDied","Data":"679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009"}
Nov 02 15:52:30.168645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.168570    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv" event={"ID":"66419e73-7d9e-45f8-bf42-d38f673f4d7b","Type":"ContainerDied","Data":"8a33a269daaf9c25ce8379def879afb69e279f0c502104b8a6fdce8b557df586"}
Nov 02 15:52:30.168645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.168496    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"
Nov 02 15:52:30.183020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.182999    5041 scope.go:122] "RemoveContainer" containerID="9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba"
Nov 02 15:52:30.183325 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:30.183301    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba\": container with ID starting with 9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba not found: ID does not exist" containerID="9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba"
Nov 02 15:52:30.183407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.183331    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba"} err="failed to get container status \"9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba\": rpc error: code = NotFound desc = could not find container \"9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba\": container with ID starting with 9582225d2e2834b2ca67e31cd84516e985093a074859eadf582f50a92d61deba not found: ID does not exist"
Nov 02 15:52:30.183407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.183348    5041 scope.go:122] "RemoveContainer" containerID="679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009"
Nov 02 15:52:30.192736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.192716    5041 scope.go:122] "RemoveContainer" containerID="679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009"
Nov 02 15:52:30.192990 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:30.192974    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009\": container with ID starting with 679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009 not found: ID does not exist" containerID="679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009"
Nov 02 15:52:30.193060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.192996    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009"} err="failed to get container status \"679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009\": rpc error: code = NotFound desc = could not find container \"679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009\": container with ID starting with 679b697262001b3225bb68b17498e317db69e437adbe871e1f1a89496a361009 not found: ID does not exist"
Nov 02 15:52:30.225591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.225561    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"]
Nov 02 15:52:30.234689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.234661    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-453864-64544968ff-ldcvv"]
Nov 02 15:52:30.261833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.261808    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"]
Nov 02 15:52:30.266351 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:30.266326    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-453864-5c775fbbff-fzlbn"]
Nov 02 15:52:31.888303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:31.888273    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" path="/var/lib/kubelet/pods/66419e73-7d9e-45f8-bf42-d38f673f4d7b/volumes"
Nov 02 15:52:31.888733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:31.888716    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" path="/var/lib/kubelet/pods/e617804b-9177-4740-b61e-9ae08e8f45a0/volumes"
Nov 02 15:52:33.470232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.470192    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"]
Nov 02 15:52:33.470803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.470503    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" podUID="6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" containerName="maas-api" containerID="cri-o://1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76" gracePeriod=30
Nov 02 15:52:33.755078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.755049    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:52:33.826530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.826498    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca\") pod \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") "
Nov 02 15:52:33.826708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.826587    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\") pod \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") "
Nov 02 15:52:33.826708 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.826687    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs\") pod \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") "
Nov 02 15:52:33.826827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.826776    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx\" (UniqueName: \"kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx\") pod \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\" (UID: \"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd\") "
Nov 02 15:52:33.826944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.826920    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca" pod "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" (UID: "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:33.827424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.827402    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:33.829083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.829043    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs" pod "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" (UID: "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:33.829235 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.829212    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls" pod "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" (UID: "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:33.829428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.829411    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx" pod "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" (UID: "6bd96673-cbd1-41f3-8e48-5a2ce59d48bd"). InnerVolumeSpecName "kube-api-access-ltrtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:33.928206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.928170    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:33.928206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.928208    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:33.928508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:33.928225    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrtx\" (UniqueName: \"kubernetes.io/projected/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd-kube-api-access-ltrtx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.201816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.201782    5041 generic.go:356] "Generic (PLEG): container finished" podID="6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" containerID="1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76" exitCode=0
Nov 02 15:52:34.202032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.201822    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"
Nov 02 15:52:34.202032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.201856    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" event={"ID":"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd","Type":"ContainerDied","Data":"1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76"}
Nov 02 15:52:34.202032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.201914    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p" event={"ID":"6bd96673-cbd1-41f3-8e48-5a2ce59d48bd","Type":"ContainerDied","Data":"f8daf8b4f5a33461e25e099bbe8df3da0272b86d779432e5bd05516155732035"}
Nov 02 15:52:34.202032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.201939    5041 scope.go:122] "RemoveContainer" containerID="1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76"
Nov 02 15:52:34.216285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.216262    5041 scope.go:122] "RemoveContainer" containerID="1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76"
Nov 02 15:52:34.216635 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:34.216604    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76\": container with ID starting with 1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76 not found: ID does not exist" containerID="1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76"
Nov 02 15:52:34.216754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.216646    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76"} err="failed to get container status \"1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76\": rpc error: code = NotFound desc = could not find container \"1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76\": container with ID starting with 1ddc59179cc2dd75ef74ec2d9c35f59b7ba19e603ba4eb8cab9b32bc251e2c76 not found: ID does not exist"
Nov 02 15:52:34.272854 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.272803    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"]
Nov 02 15:52:34.275613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:34.275573    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-af33a6-bf6dcc6df-cgc9p"]
Nov 02 15:52:35.889162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:35.889127    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" path="/var/lib/kubelet/pods/6bd96673-cbd1-41f3-8e48-5a2ce59d48bd/volumes"
Nov 02 15:52:36.769119 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:36.769080    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:52:40.874444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:40.874404    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"]
Nov 02 15:52:40.875238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:40.874720    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" podUID="e077bfc1-3fc9-4262-bece-e6ada8091d48" containerName="maas-api" containerID="cri-o://7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a" gracePeriod=30
Nov 02 15:52:41.156614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.156584    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:52:41.206677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.206640    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca\") pod \"e077bfc1-3fc9-4262-bece-e6ada8091d48\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") "
Nov 02 15:52:41.206876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.206707    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms\" (UniqueName: \"kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms\") pod \"e077bfc1-3fc9-4262-bece-e6ada8091d48\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") "
Nov 02 15:52:41.206876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.206742    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\") pod \"e077bfc1-3fc9-4262-bece-e6ada8091d48\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") "
Nov 02 15:52:41.207016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.206930    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs\") pod \"e077bfc1-3fc9-4262-bece-e6ada8091d48\" (UID: \"e077bfc1-3fc9-4262-bece-e6ada8091d48\") "
Nov 02 15:52:41.207079 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.207011    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca" pod "e077bfc1-3fc9-4262-bece-e6ada8091d48" (UID: "e077bfc1-3fc9-4262-bece-e6ada8091d48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:41.207483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.207448    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077bfc1-3fc9-4262-bece-e6ada8091d48-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.209544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.209512    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms" pod "e077bfc1-3fc9-4262-bece-e6ada8091d48" (UID: "e077bfc1-3fc9-4262-bece-e6ada8091d48"). InnerVolumeSpecName "kube-api-access-7nwms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:41.209801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.209777    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs" pod "e077bfc1-3fc9-4262-bece-e6ada8091d48" (UID: "e077bfc1-3fc9-4262-bece-e6ada8091d48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:41.209984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.209957    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls" pod "e077bfc1-3fc9-4262-bece-e6ada8091d48" (UID: "e077bfc1-3fc9-4262-bece-e6ada8091d48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:41.264185 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.264153    5041 generic.go:356] "Generic (PLEG): container finished" podID="e077bfc1-3fc9-4262-bece-e6ada8091d48" containerID="7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a" exitCode=0
Nov 02 15:52:41.264384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.264212    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"
Nov 02 15:52:41.264384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.264253    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" event={"ID":"e077bfc1-3fc9-4262-bece-e6ada8091d48","Type":"ContainerDied","Data":"7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a"}
Nov 02 15:52:41.264384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.264326    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c" event={"ID":"e077bfc1-3fc9-4262-bece-e6ada8091d48","Type":"ContainerDied","Data":"6d0fece76dbeab31088b6cf107d5721df01301ecbfd8cea49660f16aba676c03"}
Nov 02 15:52:41.264384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.264358    5041 scope.go:122] "RemoveContainer" containerID="7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a"
Nov 02 15:52:41.276487 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.276468    5041 scope.go:122] "RemoveContainer" containerID="7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a"
Nov 02 15:52:41.276725 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:41.276705    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a\": container with ID starting with 7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a not found: ID does not exist" containerID="7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a"
Nov 02 15:52:41.276795 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.276735    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a"} err="failed to get container status \"7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a\": rpc error: code = NotFound desc = could not find container \"7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a\": container with ID starting with 7f785455425e33893259f68a66880ab727c57f3e4665d2ec97f28826a1ceff7a not found: ID does not exist"
Nov 02 15:52:41.308165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.308121    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.308165 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.308162    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwms\" (UniqueName: \"kubernetes.io/projected/e077bfc1-3fc9-4262-bece-e6ada8091d48-kube-api-access-7nwms\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.308317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.308178    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e077bfc1-3fc9-4262-bece-e6ada8091d48-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.316186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.316161    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"]
Nov 02 15:52:41.326436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.326411    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e40940-df56fc94b-8xq7c"]
Nov 02 15:52:41.809017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.808974    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"]
Nov 02 15:52:41.841330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.841291    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"]
Nov 02 15:52:41.892703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:41.892664    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e077bfc1-3fc9-4262-bece-e6ada8091d48" path="/var/lib/kubelet/pods/e077bfc1-3fc9-4262-bece-e6ada8091d48/volumes"
Nov 02 15:52:46.809349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:46.809301    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerName="payload-processing" containerID="cri-o://64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1" gracePeriod=25
Nov 02 15:52:46.841916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:46.841873    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerName="payload-pre-processing" containerID="cri-o://7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55" gracePeriod=25
Nov 02 15:52:49.531104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:49.529430    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"]
Nov 02 15:52:49.531104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:49.529672    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy" containerID="cri-o://59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711" gracePeriod=30
Nov 02 15:52:54.143759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.143720    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"]
Nov 02 15:52:54.805103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.805041    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:52:54.954672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954642    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954685    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954716    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954786    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954814    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954897    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.954960    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955047    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955153    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket\") pod \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\" (UID: \"9d7054c1-6dbe-4aab-8bf9-291871fba5e0\") "
Nov 02 15:52:54.955440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955045    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.955440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955179    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.955440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955091    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:54.955675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955492    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.955675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955602    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.955675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955624    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.955675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955639    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.955675 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955651    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.955967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.955786    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.957748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.957713    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "kube-api-access-6tlpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:54.958715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.958686    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:54.958715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.958706    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:54.958885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:54.958789    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy" pod "9d7054c1-6dbe-4aab-8bf9-291871fba5e0" (UID: "9d7054c1-6dbe-4aab-8bf9-291871fba5e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.057115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.056959    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.057115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.056997    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.057115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.057014    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.057115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.057030    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tlpj\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-kube-api-access-6tlpj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.057115 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.057047    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7054c1-6dbe-4aab-8bf9-291871fba5e0-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.376050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.375979    5041 generic.go:356] "Generic (PLEG): container finished" podID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerID="59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711" exitCode=0
Nov 02 15:52:55.376050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.376025    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"
Nov 02 15:52:55.376589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.376058    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" event={"ID":"9d7054c1-6dbe-4aab-8bf9-291871fba5e0","Type":"ContainerDied","Data":"59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711"}
Nov 02 15:52:55.376589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.376112    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9" event={"ID":"9d7054c1-6dbe-4aab-8bf9-291871fba5e0","Type":"ContainerDied","Data":"0e41379b744297c4d36b03a2334e98db442597891403bd5c51f929dd268fd56f"}
Nov 02 15:52:55.376589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.376134    5041 scope.go:122] "RemoveContainer" containerID="59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711"
Nov 02 15:52:55.389564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.389545    5041 scope.go:122] "RemoveContainer" containerID="59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711"
Nov 02 15:52:55.389835 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:52:55.389813    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711\": container with ID starting with 59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711 not found: ID does not exist" containerID="59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711"
Nov 02 15:52:55.389912 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.389842    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711"} err="failed to get container status \"59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711\": rpc error: code = NotFound desc = could not find container \"59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711\": container with ID starting with 59248c0cdfe6c66c514741dc40b59d0f30c045efff2744e826ef8e230a50d711 not found: ID does not exist"
Nov 02 15:52:55.412550 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.412525    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:52:55.438480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.438456    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"]
Nov 02 15:52:55.449515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.449493    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-wv6h9"]
Nov 02 15:52:55.888643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:55.888612    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" path="/var/lib/kubelet/pods/9d7054c1-6dbe-4aab-8bf9-291871fba5e0/volumes"
Nov 02 15:52:56.418743 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:56.418691    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:52:59.367600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:52:59.367547    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" containerName="payload-pre-processing" containerID="cri-o://52a0d19de5ce7d31f7e18bdacaf74eafe3b991335464ee16d1362cd6b10de1a5" gracePeriod=25
Nov 02 15:53:01.529574 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:01.529532    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:05.412696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:05.412655    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:06.418463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:06.418416    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:06.782306 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:06.782189    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:53:07.128086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.128058    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:53:07.155552 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.155526    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:53:07.179612 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.179574    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4\" (UniqueName: \"kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4\") pod \"50362a8d-c5af-4496-b343-b60ec54f3f0f\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") "
Nov 02 15:53:07.179784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.179691    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx\" (UniqueName: \"kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx\") pod \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") "
Nov 02 15:53:07.179863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.179817    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume\") pod \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\" (UID: \"83af6371-cf0d-4d86-9b31-16e17a05e1d2\") "
Nov 02 15:53:07.180025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.179991    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume\") pod \"50362a8d-c5af-4496-b343-b60ec54f3f0f\" (UID: \"50362a8d-c5af-4496-b343-b60ec54f3f0f\") "
Nov 02 15:53:07.180379 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.180336    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume" pod "83af6371-cf0d-4d86-9b31-16e17a05e1d2" (UID: "83af6371-cf0d-4d86-9b31-16e17a05e1d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:07.180481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.180351    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume" pod "50362a8d-c5af-4496-b343-b60ec54f3f0f" (UID: "50362a8d-c5af-4496-b343-b60ec54f3f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:07.180545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.180497    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362a8d-c5af-4496-b343-b60ec54f3f0f-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.180545 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.180513    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83af6371-cf0d-4d86-9b31-16e17a05e1d2-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.182046 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.182021    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx" pod "83af6371-cf0d-4d86-9b31-16e17a05e1d2" (UID: "83af6371-cf0d-4d86-9b31-16e17a05e1d2"). InnerVolumeSpecName "kube-api-access-k4svx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:07.182187 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.182167    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4" pod "50362a8d-c5af-4496-b343-b60ec54f3f0f" (UID: "50362a8d-c5af-4496-b343-b60ec54f3f0f"). InnerVolumeSpecName "kube-api-access-zknm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:07.281998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.281963    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4svx\" (UniqueName: \"kubernetes.io/projected/83af6371-cf0d-4d86-9b31-16e17a05e1d2-kube-api-access-k4svx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.281998 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.281994    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknm4\" (UniqueName: \"kubernetes.io/projected/50362a8d-c5af-4496-b343-b60ec54f3f0f-kube-api-access-zknm4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.469886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.469620    5041 generic.go:356] "Generic (PLEG): container finished" podID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerID="7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55" exitCode=0
Nov 02 15:53:07.469886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.469662    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" event={"ID":"83af6371-cf0d-4d86-9b31-16e17a05e1d2","Type":"ContainerDied","Data":"7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55"}
Nov 02 15:53:07.469886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.469695    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"
Nov 02 15:53:07.469886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.469728    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c" event={"ID":"83af6371-cf0d-4d86-9b31-16e17a05e1d2","Type":"ContainerDied","Data":"097df980f1a436003f5b8b560df478b053374a82d1d96e8b25f33e9f2b343b5a"}
Nov 02 15:53:07.469886 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.469754    5041 scope.go:122] "RemoveContainer" containerID="7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55"
Nov 02 15:53:07.471415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.471357    5041 generic.go:356] "Generic (PLEG): container finished" podID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerID="64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1" exitCode=0
Nov 02 15:53:07.471531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.471421    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"
Nov 02 15:53:07.471531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.471506    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" event={"ID":"50362a8d-c5af-4496-b343-b60ec54f3f0f","Type":"ContainerDied","Data":"64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1"}
Nov 02 15:53:07.471661 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.471567    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9" event={"ID":"50362a8d-c5af-4496-b343-b60ec54f3f0f","Type":"ContainerDied","Data":"ee0d7df3d32bccf21ceff000d5faf522446cac4fc7c572faafe6c238095ed7a7"}
Nov 02 15:53:07.483087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.483042    5041 scope.go:122] "RemoveContainer" containerID="7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55"
Nov 02 15:53:07.483918 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:07.483800    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55\": container with ID starting with 7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55 not found: ID does not exist" containerID="7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55"
Nov 02 15:53:07.483918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.483835    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55"} err="failed to get container status \"7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55\": rpc error: code = NotFound desc = could not find container \"7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55\": container with ID starting with 7fec3dbf214f191c0304615859910fcdaad83940ddc8b204c9b760fd67c42a55 not found: ID does not exist"
Nov 02 15:53:07.483918 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.483856    5041 scope.go:122] "RemoveContainer" containerID="64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1"
Nov 02 15:53:07.492926 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.492911    5041 scope.go:122] "RemoveContainer" containerID="64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1"
Nov 02 15:53:07.493171 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:07.493150    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1\": container with ID starting with 64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1 not found: ID does not exist" containerID="64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1"
Nov 02 15:53:07.493214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.493179    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1"} err="failed to get container status \"64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1\": rpc error: code = NotFound desc = could not find container \"64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1\": container with ID starting with 64d4ec9498f4a5745c76acd042e8702923ecb2e8917de114bdce96eb8a62b0d1 not found: ID does not exist"
Nov 02 15:53:07.557628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.557606    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"]
Nov 02 15:53:07.561949 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.561929    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6af50f-5cbbbf8747-4rvl9"]
Nov 02 15:53:07.593803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.593783    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"]
Nov 02 15:53:07.601342 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.601321    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6af50f-859b5bf6c9-h469c"]
Nov 02 15:53:07.890358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.890323    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" path="/var/lib/kubelet/pods/50362a8d-c5af-4496-b343-b60ec54f3f0f/volumes"
Nov 02 15:53:07.890749 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:07.890727    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" path="/var/lib/kubelet/pods/83af6371-cf0d-4d86-9b31-16e17a05e1d2/volumes"
Nov 02 15:53:10.485348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.485306    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"]
Nov 02 15:53:10.485811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.485615    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5" podUID="962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" containerName="maas-api" containerID="cri-o://95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5" gracePeriod=30
Nov 02 15:53:10.766265 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.766219    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"]
Nov 02 15:53:10.766597 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.766537    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" podUID="8988bd7c-4215-4f96-817f-439871ea3f75" containerName="payload-processing" containerID="cri-o://32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69" gracePeriod=30
Nov 02 15:53:10.786435 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.786381    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"]
Nov 02 15:53:10.786628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.786603    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" podUID="d27bbce5-5794-4efe-9248-0716a561918f" containerName="payload-pre-processing" containerID="cri-o://d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330" gracePeriod=30
Nov 02 15:53:10.793715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.793689    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:53:10.819842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.819817    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x\" (UniqueName: \"kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x\") pod \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") "
Nov 02 15:53:10.819947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.819889    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca\") pod \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") "
Nov 02 15:53:10.820019 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.819995    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs\") pod \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") "
Nov 02 15:53:10.820107 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.820081    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls\") pod \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\" (UID: \"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65\") "
Nov 02 15:53:10.820879 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.820847    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca" pod "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" (UID: "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:10.824180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.824156    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls" pod "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" (UID: "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:10.828325 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.828296    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x" pod "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" (UID: "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65"). InnerVolumeSpecName "kube-api-access-zjd4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:10.830084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.830054    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs" pod "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" (UID: "962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:10.921310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.921276    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjd4x\" (UniqueName: \"kubernetes.io/projected/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-kube-api-access-zjd4x\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:10.921310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.921311    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:10.921580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.921353    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:10.921580 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.921378    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:10.921705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.921660    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" podUID="d27bbce5-5794-4efe-9248-0716a561918f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.1.63:9004: connect: connection refused"
Nov 02 15:53:10.921983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:10.921950    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" podUID="8988bd7c-4215-4f96-817f-439871ea3f75" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.1.64:9004: connect: connection refused"
Nov 02 15:53:11.042816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.042785    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:53:11.089925 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.089887    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:53:11.123736 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.123697    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume\") pod \"8988bd7c-4215-4f96-817f-439871ea3f75\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") "
Nov 02 15:53:11.123937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.123814    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume\") pod \"d27bbce5-5794-4efe-9248-0716a561918f\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") "
Nov 02 15:53:11.123937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.123869    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x\" (UniqueName: \"kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x\") pod \"8988bd7c-4215-4f96-817f-439871ea3f75\" (UID: \"8988bd7c-4215-4f96-817f-439871ea3f75\") "
Nov 02 15:53:11.123937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.123907    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz\" (UniqueName: \"kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz\") pod \"d27bbce5-5794-4efe-9248-0716a561918f\" (UID: \"d27bbce5-5794-4efe-9248-0716a561918f\") "
Nov 02 15:53:11.125579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.125544    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume" pod "8988bd7c-4215-4f96-817f-439871ea3f75" (UID: "8988bd7c-4215-4f96-817f-439871ea3f75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:11.126056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.126029    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume" pod "d27bbce5-5794-4efe-9248-0716a561918f" (UID: "d27bbce5-5794-4efe-9248-0716a561918f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:11.126864 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.126834    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz" pod "d27bbce5-5794-4efe-9248-0716a561918f" (UID: "d27bbce5-5794-4efe-9248-0716a561918f"). InnerVolumeSpecName "kube-api-access-zvvbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:11.128603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.128581    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x" pod "8988bd7c-4215-4f96-817f-439871ea3f75" (UID: "8988bd7c-4215-4f96-817f-439871ea3f75"). InnerVolumeSpecName "kube-api-access-vcr6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:11.225417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.225353    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8988bd7c-4215-4f96-817f-439871ea3f75-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:11.225417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.225413    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d27bbce5-5794-4efe-9248-0716a561918f-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:11.225417 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.225423    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcr6x\" (UniqueName: \"kubernetes.io/projected/8988bd7c-4215-4f96-817f-439871ea3f75-kube-api-access-vcr6x\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:11.225642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.225432    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvbz\" (UniqueName: \"kubernetes.io/projected/d27bbce5-5794-4efe-9248-0716a561918f-kube-api-access-zvvbz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:11.505997 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.505962    5041 generic.go:356] "Generic (PLEG): container finished" podID="d27bbce5-5794-4efe-9248-0716a561918f" containerID="d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330" exitCode=0
Nov 02 15:53:11.506481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.506004    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"
Nov 02 15:53:11.506481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.506039    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" event={"ID":"d27bbce5-5794-4efe-9248-0716a561918f","Type":"ContainerDied","Data":"d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330"}
Nov 02 15:53:11.506481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.506095    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9" event={"ID":"d27bbce5-5794-4efe-9248-0716a561918f","Type":"ContainerDied","Data":"54f58f38d8c13aff9d8684eda7e471591c779067804898533f1efc969acdf078"}
Nov 02 15:53:11.506481 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.506123    5041 scope.go:122] "RemoveContainer" containerID="d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330"
Nov 02 15:53:11.508055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.508029    5041 generic.go:356] "Generic (PLEG): container finished" podID="962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" containerID="95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5" exitCode=0
Nov 02 15:53:11.508177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.508083    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"
Nov 02 15:53:11.508177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.508086    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5" event={"ID":"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65","Type":"ContainerDied","Data":"95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5"}
Nov 02 15:53:11.508177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.508151    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5" event={"ID":"962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65","Type":"ContainerDied","Data":"3dcacf7cd1797007532235829fe5a88e21a76e8d0475a7f09420e18d7196fa0f"}
Nov 02 15:53:11.516281 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.516261    5041 generic.go:356] "Generic (PLEG): container finished" podID="8988bd7c-4215-4f96-817f-439871ea3f75" containerID="32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69" exitCode=0
Nov 02 15:53:11.516406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.516328    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" event={"ID":"8988bd7c-4215-4f96-817f-439871ea3f75","Type":"ContainerDied","Data":"32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69"}
Nov 02 15:53:11.516406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.516381    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n" event={"ID":"8988bd7c-4215-4f96-817f-439871ea3f75","Type":"ContainerDied","Data":"47e95d686adcf30ccb992b76efe8f1dc1fc966a97ba826efda791710dd98ba73"}
Nov 02 15:53:11.516593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.516443    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"
Nov 02 15:53:11.529425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.529396    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:11.531706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.531689    5041 scope.go:122] "RemoveContainer" containerID="d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330"
Nov 02 15:53:11.531938 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:11.531919    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330\": container with ID starting with d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330 not found: ID does not exist" containerID="d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330"
Nov 02 15:53:11.532002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.531943    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330"} err="failed to get container status \"d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330\": rpc error: code = NotFound desc = could not find container \"d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330\": container with ID starting with d985b6c1727f585fac195e79fd92d3e27ba3f861fc764d337f63260d41408330 not found: ID does not exist"
Nov 02 15:53:11.532002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.531964    5041 scope.go:122] "RemoveContainer" containerID="95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5"
Nov 02 15:53:11.540913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.540894    5041 scope.go:122] "RemoveContainer" containerID="95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5"
Nov 02 15:53:11.541165 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:11.541146    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5\": container with ID starting with 95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5 not found: ID does not exist" containerID="95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5"
Nov 02 15:53:11.541233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.541170    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5"} err="failed to get container status \"95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5\": rpc error: code = NotFound desc = could not find container \"95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5\": container with ID starting with 95328d635bfbac83cf628d9f421aeb6a9988fb72c6feec5fab2ed35c4aaef4f5 not found: ID does not exist"
Nov 02 15:53:11.541233 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.541193    5041 scope.go:122] "RemoveContainer" containerID="32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69"
Nov 02 15:53:11.549955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.549939    5041 scope.go:122] "RemoveContainer" containerID="32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69"
Nov 02 15:53:11.550230 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:11.550212    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69\": container with ID starting with 32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69 not found: ID does not exist" containerID="32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69"
Nov 02 15:53:11.550286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.550235    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69"} err="failed to get container status \"32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69\": rpc error: code = NotFound desc = could not find container \"32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69\": container with ID starting with 32aa590cc8a109e8e12c1bbdb89aafbf1156dc681907a20016256ec4f4a5ff69 not found: ID does not exist"
Nov 02 15:53:11.616668 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.616645    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"]
Nov 02 15:53:11.643712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.643688    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a433eb-7d8585c94c-6bvz9"]
Nov 02 15:53:11.683087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.683063    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"]
Nov 02 15:53:11.710400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.710357    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a433eb-7794dd854d-ltz8n"]
Nov 02 15:53:11.746602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.746577    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"]
Nov 02 15:53:11.749463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.749443    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a433eb-56b79d5775-sp6f5"]
Nov 02 15:53:11.888246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.888180    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8988bd7c-4215-4f96-817f-439871ea3f75" path="/var/lib/kubelet/pods/8988bd7c-4215-4f96-817f-439871ea3f75/volumes"
Nov 02 15:53:11.888626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.888611    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" path="/var/lib/kubelet/pods/962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65/volumes"
Nov 02 15:53:11.888968 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:11.888956    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27bbce5-5794-4efe-9248-0716a561918f" path="/var/lib/kubelet/pods/d27bbce5-5794-4efe-9248-0716a561918f/volumes"
Nov 02 15:53:19.581469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.581432    5041 generic.go:356] "Generic (PLEG): container finished" podID="046d2093-4395-4dc8-9a50-cdba9e799355" containerID="52a0d19de5ce7d31f7e18bdacaf74eafe3b991335464ee16d1362cd6b10de1a5" exitCode=0
Nov 02 15:53:19.581857 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.581502    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" event={"ID":"046d2093-4395-4dc8-9a50-cdba9e799355","Type":"ContainerDied","Data":"52a0d19de5ce7d31f7e18bdacaf74eafe3b991335464ee16d1362cd6b10de1a5"}
Nov 02 15:53:19.628106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.628056    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:53:19.722383 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.722343    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume\") pod \"046d2093-4395-4dc8-9a50-cdba9e799355\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") "
Nov 02 15:53:19.722548 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.722478    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv\" (UniqueName: \"kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv\") pod \"046d2093-4395-4dc8-9a50-cdba9e799355\" (UID: \"046d2093-4395-4dc8-9a50-cdba9e799355\") "
Nov 02 15:53:19.722710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.722684    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume" pod "046d2093-4395-4dc8-9a50-cdba9e799355" (UID: "046d2093-4395-4dc8-9a50-cdba9e799355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:19.722920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.722905    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046d2093-4395-4dc8-9a50-cdba9e799355-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:19.724570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.724548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv" pod "046d2093-4395-4dc8-9a50-cdba9e799355" (UID: "046d2093-4395-4dc8-9a50-cdba9e799355"). InnerVolumeSpecName "kube-api-access-pjdmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:19.824020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:19.823995    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdmv\" (UniqueName: \"kubernetes.io/projected/046d2093-4395-4dc8-9a50-cdba9e799355-kube-api-access-pjdmv\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:20.589947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:20.589918    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"
Nov 02 15:53:20.590449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:20.589950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb" event={"ID":"046d2093-4395-4dc8-9a50-cdba9e799355","Type":"ContainerDied","Data":"b7b071d3439efc0b4216c7dfd6b7c13b87b2d2ea8c28bea6e694a1618c4f8da8"}
Nov 02 15:53:20.590449 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:20.590020    5041 scope.go:122] "RemoveContainer" containerID="52a0d19de5ce7d31f7e18bdacaf74eafe3b991335464ee16d1362cd6b10de1a5"
Nov 02 15:53:20.644356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:20.644327    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"]
Nov 02 15:53:20.653226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:20.653205    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6f2e0c-694bc6bfff-2qpqb"]
Nov 02 15:53:21.889805 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:21.889763    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" path="/var/lib/kubelet/pods/046d2093-4395-4dc8-9a50-cdba9e799355/volumes"
Nov 02 15:53:22.047837 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.047805    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"]
Nov 02 15:53:22.090647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.090614    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"]
Nov 02 15:53:22.338807 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.338771    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"]
Nov 02 15:53:22.339050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.339021    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" podUID="2fcc6398-b336-4fc4-8a97-0add8ca0a52d" containerName="maas-api" containerID="cri-o://8449e770b0c7488ca896823bd5e15026597db2b3cdb0539581846b4af0def555" gracePeriod=30
Nov 02 15:53:22.606499 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.606428    5041 generic.go:356] "Generic (PLEG): container finished" podID="2fcc6398-b336-4fc4-8a97-0add8ca0a52d" containerID="8449e770b0c7488ca896823bd5e15026597db2b3cdb0539581846b4af0def555" exitCode=0
Nov 02 15:53:22.606615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.606507    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" event={"ID":"2fcc6398-b336-4fc4-8a97-0add8ca0a52d","Type":"ContainerDied","Data":"8449e770b0c7488ca896823bd5e15026597db2b3cdb0539581846b4af0def555"}
Nov 02 15:53:22.606615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.606559    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:53:22.606715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.606561    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms" event={"ID":"2fcc6398-b336-4fc4-8a97-0add8ca0a52d","Type":"ContainerDied","Data":"403d1b9eed52660b91f2c163db1914694ee31e7d3b1be75c56fce0cefd3d9915"}
Nov 02 15:53:22.606715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.606675    5041 scope.go:122] "RemoveContainer" containerID="8449e770b0c7488ca896823bd5e15026597db2b3cdb0539581846b4af0def555"
Nov 02 15:53:22.756701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.756661    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb\" (UniqueName: \"kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb\") pod \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") "
Nov 02 15:53:22.756905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.756710    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs\") pod \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") "
Nov 02 15:53:22.756905 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.756766    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca\") pod \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") "
Nov 02 15:53:22.757022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.756929    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\") pod \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\" (UID: \"2fcc6398-b336-4fc4-8a97-0add8ca0a52d\") "
Nov 02 15:53:22.757209 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.757170    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca" pod "2fcc6398-b336-4fc4-8a97-0add8ca0a52d" (UID: "2fcc6398-b336-4fc4-8a97-0add8ca0a52d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:22.757469 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.757425    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:22.758973 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.758942    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs" pod "2fcc6398-b336-4fc4-8a97-0add8ca0a52d" (UID: "2fcc6398-b336-4fc4-8a97-0add8ca0a52d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:22.759266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.759239    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls" pod "2fcc6398-b336-4fc4-8a97-0add8ca0a52d" (UID: "2fcc6398-b336-4fc4-8a97-0add8ca0a52d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:22.759388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.759276    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb" pod "2fcc6398-b336-4fc4-8a97-0add8ca0a52d" (UID: "2fcc6398-b336-4fc4-8a97-0add8ca0a52d"). InnerVolumeSpecName "kube-api-access-m2pcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:22.858704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.858642    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:22.858704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.858668    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2pcb\" (UniqueName: \"kubernetes.io/projected/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-kube-api-access-m2pcb\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:22.858704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:22.858680    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcc6398-b336-4fc4-8a97-0add8ca0a52d-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.614815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:23.614781    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"
Nov 02 15:53:23.670413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:23.670377    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"]
Nov 02 15:53:23.681515 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:23.681492    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-797488-5ffdc84cc6-49rms"]
Nov 02 15:53:23.887794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:23.887714    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcc6398-b336-4fc4-8a97-0add8ca0a52d" path="/var/lib/kubelet/pods/2fcc6398-b336-4fc4-8a97-0add8ca0a52d/volumes"
Nov 02 15:53:27.048755 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:27.048712    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerName="payload-processing" containerID="cri-o://b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348" gracePeriod=25
Nov 02 15:53:27.091690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:27.091655    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerName="payload-pre-processing" containerID="cri-o://400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1" gracePeriod=25
Nov 02 15:53:28.185768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:28.185732    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:28.186206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:28.185789    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:35.946759 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:35.946723    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"]
Nov 02 15:53:35.947262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:35.946939    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2" podUID="e4dd368d-0046-43ab-9fa6-0edbffa602c8" containerName="maas-api" containerID="cri-o://3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7" gracePeriod=30
Nov 02 15:53:36.240707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.240678    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:53:36.293703 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.293666    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls\") pod \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") "
Nov 02 15:53:36.293844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.293827    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca\") pod \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") "
Nov 02 15:53:36.293921 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.293884    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh\" (UniqueName: \"kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh\") pod \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") "
Nov 02 15:53:36.294317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.294267    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca" pod "e4dd368d-0046-43ab-9fa6-0edbffa602c8" (UID: "e4dd368d-0046-43ab-9fa6-0edbffa602c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:36.296214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.296190    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh" pod "e4dd368d-0046-43ab-9fa6-0edbffa602c8" (UID: "e4dd368d-0046-43ab-9fa6-0edbffa602c8"). InnerVolumeSpecName "kube-api-access-rrmzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:36.296214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.296203    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls" pod "e4dd368d-0046-43ab-9fa6-0edbffa602c8" (UID: "e4dd368d-0046-43ab-9fa6-0edbffa602c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:36.394788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.394754    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs\") pod \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\" (UID: \"e4dd368d-0046-43ab-9fa6-0edbffa602c8\") "
Nov 02 15:53:36.395258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.395238    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:36.395376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.395263    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd368d-0046-43ab-9fa6-0edbffa602c8-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:36.395376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.395277    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmzh\" (UniqueName: \"kubernetes.io/projected/e4dd368d-0046-43ab-9fa6-0edbffa602c8-kube-api-access-rrmzh\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:36.396823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.396795    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs" pod "e4dd368d-0046-43ab-9fa6-0edbffa602c8" (UID: "e4dd368d-0046-43ab-9fa6-0edbffa602c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:36.496172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.496088    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4dd368d-0046-43ab-9fa6-0edbffa602c8-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:36.712989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.712951    5041 generic.go:356] "Generic (PLEG): container finished" podID="e4dd368d-0046-43ab-9fa6-0edbffa602c8" containerID="3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7" exitCode=0
Nov 02 15:53:36.712989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.712989    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"
Nov 02 15:53:36.713255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.712998    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2" event={"ID":"e4dd368d-0046-43ab-9fa6-0edbffa602c8","Type":"ContainerDied","Data":"3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7"}
Nov 02 15:53:36.713255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.713048    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2" event={"ID":"e4dd368d-0046-43ab-9fa6-0edbffa602c8","Type":"ContainerDied","Data":"8d933fd909563089c2bd9e8011ab80cc90b4fb3d158f91ab69cc0eaa9d713146"}
Nov 02 15:53:36.713255 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.713071    5041 scope.go:122] "RemoveContainer" containerID="3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7"
Nov 02 15:53:36.724748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.724726    5041 scope.go:122] "RemoveContainer" containerID="3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7"
Nov 02 15:53:36.725083 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:36.725053    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7\": container with ID starting with 3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7 not found: ID does not exist" containerID="3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7"
Nov 02 15:53:36.725166 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.725091    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7"} err="failed to get container status \"3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7\": rpc error: code = NotFound desc = could not find container \"3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7\": container with ID starting with 3c3920e7badf480c08a1b8b521b864fd56c33b4e14d682e43b16e87cf2e899e7 not found: ID does not exist"
Nov 02 15:53:36.771335 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:36.771258    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:53:36.787129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.787092    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"]
Nov 02 15:53:36.790231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:36.790208    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f6f1a0-6c589c4b6c-xjrt2"]
Nov 02 15:53:37.889117 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:37.889084    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4dd368d-0046-43ab-9fa6-0edbffa602c8" path="/var/lib/kubelet/pods/e4dd368d-0046-43ab-9fa6-0edbffa602c8/volumes"
Nov 02 15:53:38.186592 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:38.186557    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:38.186734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:38.186567    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:38.882853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:38.882822    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"]
Nov 02 15:53:38.894573 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:38.894545    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"]
Nov 02 15:53:44.721706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:44.721671    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"]
Nov 02 15:53:44.773634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:44.773602    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"]
Nov 02 15:53:47.364002 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.363974    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:53:47.397192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.393792    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9\" (UniqueName: \"kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9\") pod \"8d3fa945-454d-4a0d-8584-13749fbc0e71\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") "
Nov 02 15:53:47.397192 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.393908    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume\") pod \"8d3fa945-454d-4a0d-8584-13749fbc0e71\" (UID: \"8d3fa945-454d-4a0d-8584-13749fbc0e71\") "
Nov 02 15:53:47.397644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.397612    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume" pod "8d3fa945-454d-4a0d-8584-13749fbc0e71" (UID: "8d3fa945-454d-4a0d-8584-13749fbc0e71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:47.399172 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.399148    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9" pod "8d3fa945-454d-4a0d-8584-13749fbc0e71" (UID: "8d3fa945-454d-4a0d-8584-13749fbc0e71"). InnerVolumeSpecName "kube-api-access-cdsk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:47.401014 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.400992    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:53:47.495272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.495244    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume\") pod \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") "
Nov 02 15:53:47.495483 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.495319    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx\" (UniqueName: \"kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx\") pod \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\" (UID: \"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3\") "
Nov 02 15:53:47.495672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.495649    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume" pod "d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" (UID: "d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:47.495750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.495701    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdsk9\" (UniqueName: \"kubernetes.io/projected/8d3fa945-454d-4a0d-8584-13749fbc0e71-kube-api-access-cdsk9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:47.495750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.495729    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3fa945-454d-4a0d-8584-13749fbc0e71-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:47.497603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.497579    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx" pod "d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" (UID: "d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3"). InnerVolumeSpecName "kube-api-access-dqcmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:47.597310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.597232    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:47.597310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.597263    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqcmx\" (UniqueName: \"kubernetes.io/projected/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3-kube-api-access-dqcmx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:47.791525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.791488    5041 generic.go:356] "Generic (PLEG): container finished" podID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerID="b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348" exitCode=0
Nov 02 15:53:47.791706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.791549    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"
Nov 02 15:53:47.791706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.791558    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" event={"ID":"8d3fa945-454d-4a0d-8584-13749fbc0e71","Type":"ContainerDied","Data":"b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348"}
Nov 02 15:53:47.791706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.791615    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw" event={"ID":"8d3fa945-454d-4a0d-8584-13749fbc0e71","Type":"ContainerDied","Data":"df7e6359f83b200957d35d30a64d48598175888fecb28b92045ada0841e3df24"}
Nov 02 15:53:47.791706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.791647    5041 scope.go:122] "RemoveContainer" containerID="b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348"
Nov 02 15:53:47.793282 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.793258    5041 generic.go:356] "Generic (PLEG): container finished" podID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerID="400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1" exitCode=0
Nov 02 15:53:47.793440 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.793422    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"
Nov 02 15:53:47.793532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.793517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" event={"ID":"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3","Type":"ContainerDied","Data":"400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1"}
Nov 02 15:53:47.793600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.793553    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5" event={"ID":"d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3","Type":"ContainerDied","Data":"059f23caabfbeaa210f8180cf2b000cce15a3bc1d07d15e3d8946151e5c7c02b"}
Nov 02 15:53:47.807498 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.807477    5041 scope.go:122] "RemoveContainer" containerID="b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348"
Nov 02 15:53:47.808032 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:47.807998    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348\": container with ID starting with b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348 not found: ID does not exist" containerID="b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348"
Nov 02 15:53:47.808120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.808039    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348"} err="failed to get container status \"b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348\": rpc error: code = NotFound desc = could not find container \"b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348\": container with ID starting with b91573cd8912a3a6609ff0ad42f172427548693339422972fa01e749843b1348 not found: ID does not exist"
Nov 02 15:53:47.808120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.808057    5041 scope.go:122] "RemoveContainer" containerID="400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1"
Nov 02 15:53:47.818120 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.818101    5041 scope.go:122] "RemoveContainer" containerID="400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1"
Nov 02 15:53:47.818379 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:47.818343    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1\": container with ID starting with 400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1 not found: ID does not exist" containerID="400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1"
Nov 02 15:53:47.818461 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.818381    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1"} err="failed to get container status \"400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1\": rpc error: code = NotFound desc = could not find container \"400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1\": container with ID starting with 400aa514742f66aa57a09ff875a44af1a542811dd42005a62f88e6ebaf4bfca1 not found: ID does not exist"
Nov 02 15:53:47.852858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.852809    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"]
Nov 02 15:53:47.856470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.856450    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-797488-78d8f775fc-pfwsw"]
Nov 02 15:53:47.887354 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.887326    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" path="/var/lib/kubelet/pods/8d3fa945-454d-4a0d-8584-13749fbc0e71/volumes"
Nov 02 15:53:47.890163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.890140    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"]
Nov 02 15:53:47.893802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:47.893781    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-797488-747856f9dc-xzrb5"]
Nov 02 15:53:49.722797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:49.722756    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerName="payload-processing" containerID="cri-o://5239776d46447a2ef17e2dc061555d87046f4b4abee3b14c868a0bb6c00a4277" gracePeriod=25
Nov 02 15:53:49.774933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:49.774896    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerName="payload-pre-processing" containerID="cri-o://61fc4f54ee902185613e82743d47b91f04f0ceaef181c7384fb647f615bd3e78" gracePeriod=25
Nov 02 15:53:49.887869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:49.887829    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" path="/var/lib/kubelet/pods/d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3/volumes"
Nov 02 15:53:53.904218 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:53.904183    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main" containerID="cri-o://17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303" gracePeriod=45
Nov 02 15:53:53.914652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:53.914622    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" containerID="cri-o://a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37" gracePeriod=45
Nov 02 15:53:54.112104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.112062    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused"
Nov 02 15:53:54.176088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.175740    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:53:54.216040 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.216017    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:53:54.268450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.268417    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache\") pod \"9fff81be-b147-4d7c-b61f-0da2100e237b\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") "
Nov 02 15:53:54.268750 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.268721    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir\") pod \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") "
Nov 02 15:53:54.268920 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.268754    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache" pod "9fff81be-b147-4d7c-b61f-0da2100e237b" (UID: "9fff81be-b147-4d7c-b61f-0da2100e237b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.268983 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.268952    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir" pod "57bd7d06-b376-4dde-8a5e-f5d1149f83b2" (UID: "57bd7d06-b376-4dde-8a5e-f5d1149f83b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.269105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269081    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm\") pod \"9fff81be-b147-4d7c-b61f-0da2100e237b\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") "
Nov 02 15:53:54.269221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269138    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir\") pod \"9fff81be-b147-4d7c-b61f-0da2100e237b\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") "
Nov 02 15:53:54.269221 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269181    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6\" (UniqueName: \"kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6\") pod \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") "
Nov 02 15:53:54.269411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269250    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs\") pod \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") "
Nov 02 15:53:54.269411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269279    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home\") pod \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") "
Nov 02 15:53:54.269411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269304    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm\") pod \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") "
Nov 02 15:53:54.269411 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269357    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs\") pod \"9fff81be-b147-4d7c-b61f-0da2100e237b\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") "
Nov 02 15:53:54.269642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269413    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6\" (UniqueName: \"kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6\") pod \"9fff81be-b147-4d7c-b61f-0da2100e237b\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") "
Nov 02 15:53:54.269642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269446    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir" pod "9fff81be-b147-4d7c-b61f-0da2100e237b" (UID: "9fff81be-b147-4d7c-b61f-0da2100e237b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.269642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269467    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home\") pod \"9fff81be-b147-4d7c-b61f-0da2100e237b\" (UID: \"9fff81be-b147-4d7c-b61f-0da2100e237b\") "
Nov 02 15:53:54.269642 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269503    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache\") pod \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\" (UID: \"57bd7d06-b376-4dde-8a5e-f5d1149f83b2\") "
Nov 02 15:53:54.269866 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.269656    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home" pod "57bd7d06-b376-4dde-8a5e-f5d1149f83b2" (UID: "57bd7d06-b376-4dde-8a5e-f5d1149f83b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.270053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.270032    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.270134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.270056    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.270134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.270065    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.270134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.270077    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.270134 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.270109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache" pod "57bd7d06-b376-4dde-8a5e-f5d1149f83b2" (UID: "57bd7d06-b376-4dde-8a5e-f5d1149f83b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.271424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.270865    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home" pod "9fff81be-b147-4d7c-b61f-0da2100e237b" (UID: "9fff81be-b147-4d7c-b61f-0da2100e237b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.271695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.271664    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm" pod "9fff81be-b147-4d7c-b61f-0da2100e237b" (UID: "9fff81be-b147-4d7c-b61f-0da2100e237b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.272689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.272666    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs" pod "57bd7d06-b376-4dde-8a5e-f5d1149f83b2" (UID: "57bd7d06-b376-4dde-8a5e-f5d1149f83b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:54.272811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.272716    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6" pod "57bd7d06-b376-4dde-8a5e-f5d1149f83b2" (UID: "57bd7d06-b376-4dde-8a5e-f5d1149f83b2"). InnerVolumeSpecName "kube-api-access-zm5l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:54.272938 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.272918    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs" pod "9fff81be-b147-4d7c-b61f-0da2100e237b" (UID: "9fff81be-b147-4d7c-b61f-0da2100e237b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:54.273150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.273123    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6" pod "9fff81be-b147-4d7c-b61f-0da2100e237b" (UID: "9fff81be-b147-4d7c-b61f-0da2100e237b"). InnerVolumeSpecName "kube-api-access-gw6t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:54.273916 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.273888    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm" pod "57bd7d06-b376-4dde-8a5e-f5d1149f83b2" (UID: "57bd7d06-b376-4dde-8a5e-f5d1149f83b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.371701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371668    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371697    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5l6\" (UniqueName: \"kubernetes.io/projected/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-kube-api-access-zm5l6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371709    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371718    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371735    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fff81be-b147-4d7c-b61f-0da2100e237b-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371743    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6t6\" (UniqueName: \"kubernetes.io/projected/9fff81be-b147-4d7c-b61f-0da2100e237b-kube-api-access-gw6t6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371756    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fff81be-b147-4d7c-b61f-0da2100e237b-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.371896 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.371770    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bd7d06-b376-4dde-8a5e-f5d1149f83b2-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.846024 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.845989    5041 generic.go:356] "Generic (PLEG): container finished" podID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerID="17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303" exitCode=0
Nov 02 15:53:54.846230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.846037    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"
Nov 02 15:53:54.846230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.846078    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" event={"ID":"57bd7d06-b376-4dde-8a5e-f5d1149f83b2","Type":"ContainerDied","Data":"17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303"}
Nov 02 15:53:54.846230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.846127    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg" event={"ID":"57bd7d06-b376-4dde-8a5e-f5d1149f83b2","Type":"ContainerDied","Data":"1ddaaa2bb00dd3aff5597bb9c41d068687c41323fdd4215c99b4cce52dc9b859"}
Nov 02 15:53:54.846230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.846152    5041 scope.go:122] "RemoveContainer" containerID="17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303"
Nov 02 15:53:54.847549 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.847524    5041 generic.go:356] "Generic (PLEG): container finished" podID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerID="a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37" exitCode=0
Nov 02 15:53:54.847665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.847564    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"
Nov 02 15:53:54.847665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.847575    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" event={"ID":"9fff81be-b147-4d7c-b61f-0da2100e237b","Type":"ContainerDied","Data":"a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37"}
Nov 02 15:53:54.847665 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.847603    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp" event={"ID":"9fff81be-b147-4d7c-b61f-0da2100e237b","Type":"ContainerDied","Data":"b0a2d78077ec1d941df6810d7ee7115e96ea0cce7d380217b380cc929be50e83"}
Nov 02 15:53:54.866402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.862837    5041 scope.go:122] "RemoveContainer" containerID="17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303"
Nov 02 15:53:54.866402 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:54.863296    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303\": container with ID starting with 17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303 not found: ID does not exist" containerID="17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303"
Nov 02 15:53:54.866402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.863336    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303"} err="failed to get container status \"17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303\": rpc error: code = NotFound desc = could not find container \"17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303\": container with ID starting with 17e57f22a5ca2896fc70442d7e1ee4ed27cf3177b1da28dd16e89bed093e1303 not found: ID does not exist"
Nov 02 15:53:54.866402 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.863379    5041 scope.go:122] "RemoveContainer" containerID="a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37"
Nov 02 15:53:54.880638 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.880617    5041 scope.go:122] "RemoveContainer" containerID="a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37"
Nov 02 15:53:54.880891 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:53:54.880871    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37\": container with ID starting with a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37 not found: ID does not exist" containerID="a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37"
Nov 02 15:53:54.880937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.880897    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37"} err="failed to get container status \"a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37\": rpc error: code = NotFound desc = could not find container \"a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37\": container with ID starting with a7ce273a72ba4d5cc6232573f3e71404d65013e3c9cd083a320b4f414a123c37 not found: ID does not exist"
Nov 02 15:53:54.925578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.925556    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"]
Nov 02 15:53:54.936472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.936452    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-w2-739bf1-kserve-74d6cf6f9-42rwg"]
Nov 02 15:53:54.966272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.966246    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"]
Nov 02 15:53:54.971414 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:54.971394    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/premium-w2-739bf1-kserve-6c597bbf56-lcrvp"]
Nov 02 15:53:55.887986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:55.887953    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" path="/var/lib/kubelet/pods/57bd7d06-b376-4dde-8a5e-f5d1149f83b2/volumes"
Nov 02 15:53:55.888385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:55.888352    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" path="/var/lib/kubelet/pods/9fff81be-b147-4d7c-b61f-0da2100e237b/volumes"
Nov 02 15:53:56.418502 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:56.418462    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:56.418944 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:56.418462    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:56.735980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:56.735895    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"]
Nov 02 15:53:56.792037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:56.792007    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"]
Nov 02 15:53:57.697424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:57.697385    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"]
Nov 02 15:53:57.698676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:57.697631    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t" podUID="027e6fd4-3b11-45ce-a79e-6a08849f7005" containerName="maas-api" containerID="cri-o://b9b86dbe00ad6f3c50ca7653fc7da88ba221ce952eca52b64a719cb45e3eabae" gracePeriod=30
Nov 02 15:53:57.871050 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:57.871018    5041 generic.go:356] "Generic (PLEG): container finished" podID="027e6fd4-3b11-45ce-a79e-6a08849f7005" containerID="b9b86dbe00ad6f3c50ca7653fc7da88ba221ce952eca52b64a719cb45e3eabae" exitCode=0
Nov 02 15:53:57.871219 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:57.871094    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t" event={"ID":"027e6fd4-3b11-45ce-a79e-6a08849f7005","Type":"ContainerDied","Data":"b9b86dbe00ad6f3c50ca7653fc7da88ba221ce952eca52b64a719cb45e3eabae"}
Nov 02 15:53:57.971961 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:57.971929    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:53:58.104652 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.104619    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs\") pod \"027e6fd4-3b11-45ce-a79e-6a08849f7005\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") "
Nov 02 15:53:58.104823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.104720    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh\" (UniqueName: \"kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh\") pod \"027e6fd4-3b11-45ce-a79e-6a08849f7005\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") "
Nov 02 15:53:58.104823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.104749    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca\") pod \"027e6fd4-3b11-45ce-a79e-6a08849f7005\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") "
Nov 02 15:53:58.104823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.104777    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls\") pod \"027e6fd4-3b11-45ce-a79e-6a08849f7005\" (UID: \"027e6fd4-3b11-45ce-a79e-6a08849f7005\") "
Nov 02 15:53:58.105119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.105095    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca" pod "027e6fd4-3b11-45ce-a79e-6a08849f7005" (UID: "027e6fd4-3b11-45ce-a79e-6a08849f7005"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:58.106960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.106929    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls" pod "027e6fd4-3b11-45ce-a79e-6a08849f7005" (UID: "027e6fd4-3b11-45ce-a79e-6a08849f7005"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:58.107088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.106956    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs" pod "027e6fd4-3b11-45ce-a79e-6a08849f7005" (UID: "027e6fd4-3b11-45ce-a79e-6a08849f7005"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:58.107088 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.106958    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh" pod "027e6fd4-3b11-45ce-a79e-6a08849f7005" (UID: "027e6fd4-3b11-45ce-a79e-6a08849f7005"). InnerVolumeSpecName "kube-api-access-5fgzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:58.205927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.205852    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgzh\" (UniqueName: \"kubernetes.io/projected/027e6fd4-3b11-45ce-a79e-6a08849f7005-kube-api-access-5fgzh\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:58.205927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.205891    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027e6fd4-3b11-45ce-a79e-6a08849f7005-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:58.205927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.205908    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:58.205927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.205921    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027e6fd4-3b11-45ce-a79e-6a08849f7005-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:53:58.275409 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.275355    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"]
Nov 02 15:53:58.276090 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276067    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276092    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276105    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276110    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276123    5041 cpu_manager.go:414] "containerMap: removing container" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerName="payload-pre-processing"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276129    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerName="payload-pre-processing"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276139    5041 cpu_manager.go:414] "containerMap: removing container" podUID="962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" containerName="maas-api"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276144    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" containerName="maas-api"
Nov 02 15:53:58.276153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276151    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276158    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276170    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8988bd7c-4215-4f96-817f-439871ea3f75" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276175    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8988bd7c-4215-4f96-817f-439871ea3f75" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276186    5041 cpu_manager.go:414] "containerMap: removing container" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276192    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276198    5041 cpu_manager.go:414] "containerMap: removing container" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276203    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276210    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276214    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276221    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276226    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276238    5041 cpu_manager.go:414] "containerMap: removing container" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276243    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276253    5041 cpu_manager.go:414] "containerMap: removing container" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276257    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276264    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276268    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276275    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6e0714ad-acf9-4929-a875-81c3b757d8b4" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276280    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0714ad-acf9-4929-a875-81c3b757d8b4" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276289    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2fcc6398-b336-4fc4-8a97-0add8ca0a52d" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276293    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcc6398-b336-4fc4-8a97-0add8ca0a52d" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276301    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276307    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276315    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276320    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276327    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276332    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276339    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e077bfc1-3fc9-4262-bece-e6ada8091d48" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276343    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e077bfc1-3fc9-4262-bece-e6ada8091d48" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276358    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e4dd368d-0046-43ab-9fa6-0edbffa602c8" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276395    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dd368d-0046-43ab-9fa6-0edbffa602c8" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276403    5041 cpu_manager.go:414] "containerMap: removing container" podUID="027e6fd4-3b11-45ce-a79e-6a08849f7005" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276408    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027e6fd4-3b11-45ce-a79e-6a08849f7005" containerName="maas-api"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276416    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276422    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerName="payload-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276429    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d27bbce5-5794-4efe-9248-0716a561918f" containerName="payload-pre-processing"
Nov 02 15:53:58.276432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276435    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27bbce5-5794-4efe-9248-0716a561918f" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276549    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="046d2093-4395-4dc8-9a50-cdba9e799355" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276559    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="50362a8d-c5af-4496-b343-b60ec54f3f0f" containerName="payload-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276565    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7054c1-6dbe-4aab-8bf9-291871fba5e0" containerName="istio-proxy"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276570    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0714ad-acf9-4929-a875-81c3b757d8b4" containerName="maas-api"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276578    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcc6398-b336-4fc4-8a97-0add8ca0a52d" containerName="maas-api"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276583    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5eb9e3-d415-402a-b228-71fc41dd80fe" containerName="istio-proxy"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276589    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3fa945-454d-4a0d-8584-13749fbc0e71" containerName="payload-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276595    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e617804b-9177-4740-b61e-9ae08e8f45a0" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276601    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e077bfc1-3fc9-4262-bece-e6ada8091d48" containerName="maas-api"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276608    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dd368d-0046-43ab-9fa6-0edbffa602c8" containerName="maas-api"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276614    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="027e6fd4-3b11-45ce-a79e-6a08849f7005" containerName="maas-api"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276620    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5704f35f-3ee9-4a3a-96b9-9d659d474ee7" containerName="payload-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276626    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27bbce5-5794-4efe-9248-0716a561918f" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276632    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58e3d37-fc86-46dd-8cb4-fab66bca973d" containerName="istio-proxy"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276637    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fff81be-b147-4d7c-b61f-0da2100e237b" containerName="main"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276645    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="83af6371-cf0d-4d86-9b31-16e17a05e1d2" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276650    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="962bcb86-ff57-4c1f-a3f9-7fe06f4a0f65" containerName="maas-api"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276655    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d365b9f3-ecd2-4eb2-a0d1-8b350288b8b3" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276662    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8988bd7c-4215-4f96-817f-439871ea3f75" containerName="payload-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276668    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="66419e73-7d9e-45f8-bf42-d38f673f4d7b" containerName="payload-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276674    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bd7d06-b376-4dde-8a5e-f5d1149f83b2" containerName="main"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276678    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5b4e77-a767-424d-9626-b2e877f1ae7b" containerName="payload-pre-processing"
Nov 02 15:53:58.277816 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.276684    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd96673-cbd1-41f3-8e48-5a2ce59d48bd" containerName="maas-api"
Nov 02 15:53:58.280377 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.280341    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.303168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.303149    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:53:58.303278 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.303154    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:53:58.310579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.310554    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"]
Nov 02 15:53:58.408400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.408345    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.408577 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.408475    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ndc\" (UniqueName: \"kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.408648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.408611    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.509486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.509415    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.509621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.509484    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ndc\" (UniqueName: \"kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.509621 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.509583    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.510225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.510205    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.511684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.511666    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.536133 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.536109    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ndc\" (UniqueName: \"kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc\") pod \"maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.595078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.595054    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:53:58.880214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.880141    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"
Nov 02 15:53:58.880214 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.880188    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t" event={"ID":"027e6fd4-3b11-45ce-a79e-6a08849f7005","Type":"ContainerDied","Data":"9d4dbe686e361912f81e2d5df37dddd25858c301c53947b6c27c4b6f55b2200b"}
Nov 02 15:53:58.880772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.880239    5041 scope.go:122] "RemoveContainer" containerID="b9b86dbe00ad6f3c50ca7653fc7da88ba221ce952eca52b64a719cb45e3eabae"
Nov 02 15:53:58.900376 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.900342    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"]
Nov 02 15:53:58.901147 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:53:58.901125    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36c184d_bc95_4465_abef_fd5e37271d75.slice/crio-756377012d0774036b6f18f9a42a17f3c972638faabdf15c298ca80ba7b9fb1e WatchSource:0}: Error finding container 756377012d0774036b6f18f9a42a17f3c972638faabdf15c298ca80ba7b9fb1e: Status 404 returned error can't find the container with id 756377012d0774036b6f18f9a42a17f3c972638faabdf15c298ca80ba7b9fb1e
Nov 02 15:53:58.942466 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.942443    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"]
Nov 02 15:53:58.946113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:58.946091    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-28da58-8647775894-s6l2t"]
Nov 02 15:53:59.889808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:59.889777    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027e6fd4-3b11-45ce-a79e-6a08849f7005" path="/var/lib/kubelet/pods/027e6fd4-3b11-45ce-a79e-6a08849f7005/volumes"
Nov 02 15:53:59.890177 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:59.890087    5041 generic.go:356] "Generic (PLEG): container finished" podID="d36c184d-bc95-4465-abef-fd5e37271d75" containerID="171d25886a9d9dff6bad8b94a538406affb331ff0a891976ac492d34e3182f72" exitCode=0
Nov 02 15:53:59.890225 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:59.890184    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw" event={"ID":"d36c184d-bc95-4465-abef-fd5e37271d75","Type":"ContainerDied","Data":"171d25886a9d9dff6bad8b94a538406affb331ff0a891976ac492d34e3182f72"}
Nov 02 15:53:59.890264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:53:59.890228    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw" event={"ID":"d36c184d-bc95-4465-abef-fd5e37271d75","Type":"ContainerStarted","Data":"756377012d0774036b6f18f9a42a17f3c972638faabdf15c298ca80ba7b9fb1e"}
Nov 02 15:54:01.041102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.041077    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:54:01.134253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.134016    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca\") pod \"d36c184d-bc95-4465-abef-fd5e37271d75\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") "
Nov 02 15:54:01.134253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.134118    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc\" (UniqueName: \"kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc\") pod \"d36c184d-bc95-4465-abef-fd5e37271d75\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") "
Nov 02 15:54:01.134253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.134157    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca\") pod \"d36c184d-bc95-4465-abef-fd5e37271d75\" (UID: \"d36c184d-bc95-4465-abef-fd5e37271d75\") "
Nov 02 15:54:01.134705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.134681    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca" pod "d36c184d-bc95-4465-abef-fd5e37271d75" (UID: "d36c184d-bc95-4465-abef-fd5e37271d75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:01.136093 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.136069    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca" pod "d36c184d-bc95-4465-abef-fd5e37271d75" (UID: "d36c184d-bc95-4465-abef-fd5e37271d75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:01.136246 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.136225    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc" pod "d36c184d-bc95-4465-abef-fd5e37271d75" (UID: "d36c184d-bc95-4465-abef-fd5e37271d75"). InnerVolumeSpecName "kube-api-access-j2ndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:01.235937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.235911    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.235937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.235935    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36c184d-bc95-4465-abef-fd5e37271d75-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.236087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.235946    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ndc\" (UniqueName: \"kubernetes.io/projected/d36c184d-bc95-4465-abef-fd5e37271d75-kube-api-access-j2ndc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.736600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.736546    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerName="payload-processing" containerID="cri-o://02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00" gracePeriod=25
Nov 02 15:54:01.792712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.792679    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerName="payload-pre-processing" containerID="cri-o://8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1" gracePeriod=25
Nov 02 15:54:01.903253 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.903231    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"
Nov 02 15:54:01.903382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.903267    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw" event={"ID":"d36c184d-bc95-4465-abef-fd5e37271d75","Type":"ContainerDied","Data":"756377012d0774036b6f18f9a42a17f3c972638faabdf15c298ca80ba7b9fb1e"}
Nov 02 15:54:01.903382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:01.903314    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="756377012d0774036b6f18f9a42a17f3c972638faabdf15c298ca80ba7b9fb1e"
Nov 02 15:54:06.418937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:06.418895    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:06.418937 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:06.418930    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:06.768759 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:06.768681    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:54:08.556349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:08.556307    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:08.556748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:08.556347    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:09.974753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:09.974498    5041 generic.go:356] "Generic (PLEG): container finished" podID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerID="5239776d46447a2ef17e2dc061555d87046f4b4abee3b14c868a0bb6c00a4277" exitCode=0
Nov 02 15:54:09.974753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:09.974689    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" event={"ID":"de4307bc-148b-44b3-a773-a137aad7ca1b","Type":"ContainerDied","Data":"5239776d46447a2ef17e2dc061555d87046f4b4abee3b14c868a0bb6c00a4277"}
Nov 02 15:54:09.977356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:09.977329    5041 generic.go:356] "Generic (PLEG): container finished" podID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerID="61fc4f54ee902185613e82743d47b91f04f0ceaef181c7384fb647f615bd3e78" exitCode=0
Nov 02 15:54:09.977524 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:09.977485    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" event={"ID":"2aa3eea8-a2a6-4608-b93e-dfce5eec5034","Type":"ContainerDied","Data":"61fc4f54ee902185613e82743d47b91f04f0ceaef181c7384fb647f615bd3e78"}
Nov 02 15:54:10.061587 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.061555    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:54:10.065020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.064995    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:54:10.223832 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.223795    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume\") pod \"de4307bc-148b-44b3-a773-a137aad7ca1b\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") "
Nov 02 15:54:10.224003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.223894    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr\") pod \"de4307bc-148b-44b3-a773-a137aad7ca1b\" (UID: \"de4307bc-148b-44b3-a773-a137aad7ca1b\") "
Nov 02 15:54:10.224003 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.223967    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume\") pod \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") "
Nov 02 15:54:10.224153 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.224086    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr\" (UniqueName: \"kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr\") pod \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\" (UID: \"2aa3eea8-a2a6-4608-b93e-dfce5eec5034\") "
Nov 02 15:54:10.224338 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.224305    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume" pod "de4307bc-148b-44b3-a773-a137aad7ca1b" (UID: "de4307bc-148b-44b3-a773-a137aad7ca1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:10.224458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.224358    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume" pod "2aa3eea8-a2a6-4608-b93e-dfce5eec5034" (UID: "2aa3eea8-a2a6-4608-b93e-dfce5eec5034"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:10.224594 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.224572    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4307bc-148b-44b3-a773-a137aad7ca1b-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:10.224669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.224598    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:10.226067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.226038    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr" pod "de4307bc-148b-44b3-a773-a137aad7ca1b" (UID: "de4307bc-148b-44b3-a773-a137aad7ca1b"). InnerVolumeSpecName "kube-api-access-hjgmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:10.226591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.226561    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr" pod "2aa3eea8-a2a6-4608-b93e-dfce5eec5034" (UID: "2aa3eea8-a2a6-4608-b93e-dfce5eec5034"). InnerVolumeSpecName "kube-api-access-vqnwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:10.325676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.325598    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqnwr\" (UniqueName: \"kubernetes.io/projected/2aa3eea8-a2a6-4608-b93e-dfce5eec5034-kube-api-access-vqnwr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:10.325676 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.325629    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjgmr\" (UniqueName: \"kubernetes.io/projected/de4307bc-148b-44b3-a773-a137aad7ca1b-kube-api-access-hjgmr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:10.985569 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.985535    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"
Nov 02 15:54:10.986102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.985579    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq" event={"ID":"de4307bc-148b-44b3-a773-a137aad7ca1b","Type":"ContainerDied","Data":"99a8a2b912231e0bf1d13ff7081a3184c81c8211ea6975600aab46dd407e774d"}
Nov 02 15:54:10.986102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.985644    5041 scope.go:122] "RemoveContainer" containerID="5239776d46447a2ef17e2dc061555d87046f4b4abee3b14c868a0bb6c00a4277"
Nov 02 15:54:10.987484 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.987460    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"
Nov 02 15:54:10.987748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.987471    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr" event={"ID":"2aa3eea8-a2a6-4608-b93e-dfce5eec5034","Type":"ContainerDied","Data":"34957b3de1fbe0d03a0adb548b923f6ba3f52801ee9c468bbf7265075c063d20"}
Nov 02 15:54:10.997672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:10.997652    5041 scope.go:122] "RemoveContainer" containerID="61fc4f54ee902185613e82743d47b91f04f0ceaef181c7384fb647f615bd3e78"
Nov 02 15:54:11.042025 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:11.041998    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"]
Nov 02 15:54:11.052911 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:11.052883    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-739bf1-56ddd75fcf-k8vnr"]
Nov 02 15:54:11.080767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:11.080740    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"]
Nov 02 15:54:11.086231 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:11.086203    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-739bf1-85b586894c-4ncqq"]
Nov 02 15:54:11.887193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:11.887159    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" path="/var/lib/kubelet/pods/2aa3eea8-a2a6-4608-b93e-dfce5eec5034/volumes"
Nov 02 15:54:11.887576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:11.887561    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" path="/var/lib/kubelet/pods/de4307bc-148b-44b3-a773-a137aad7ca1b/volumes"
Nov 02 15:54:14.038544 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:14.038504    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"]
Nov 02 15:54:14.089037 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:14.089004    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"]
Nov 02 15:54:18.556707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.556666    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:18.557119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.556728    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:18.601407 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.601374    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"]
Nov 02 15:54:18.601650 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.601618    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj" podUID="15d6fd17-fdd7-442d-a004-96d2c4e73a3b" containerName="maas-api" containerID="cri-o://0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f" gracePeriod=30
Nov 02 15:54:18.850699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.850613    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"]
Nov 02 15:54:18.850869 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.850846    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph" podUID="1d0b63d7-ab28-495d-8a45-45379a3891e1" containerName="payload-pre-processing" containerID="cri-o://2941c40016c7f33958953876f046812be5b70f07f1ffbd6e2ca3a3856830b49d" gracePeriod=30
Nov 02 15:54:18.855110 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.855079    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"]
Nov 02 15:54:18.855335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:18.855310    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp" podUID="c22dfaed-e76a-4b46-b522-9f4a6d119018" containerName="payload-processing" containerID="cri-o://7a60564fa42616c65e96552f7b8d651f439ff6c5c3ace90e111b0db7b64e0903" gracePeriod=30
Nov 02 15:54:19.000067 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.000036    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:54:19.055486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.055407    5041 generic.go:356] "Generic (PLEG): container finished" podID="c22dfaed-e76a-4b46-b522-9f4a6d119018" containerID="7a60564fa42616c65e96552f7b8d651f439ff6c5c3ace90e111b0db7b64e0903" exitCode=0
Nov 02 15:54:19.055486 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.055482    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp" event={"ID":"c22dfaed-e76a-4b46-b522-9f4a6d119018","Type":"ContainerDied","Data":"7a60564fa42616c65e96552f7b8d651f439ff6c5c3ace90e111b0db7b64e0903"}
Nov 02 15:54:19.057945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.057306    5041 generic.go:356] "Generic (PLEG): container finished" podID="15d6fd17-fdd7-442d-a004-96d2c4e73a3b" containerID="0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f" exitCode=0
Nov 02 15:54:19.057945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.057533    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj" event={"ID":"15d6fd17-fdd7-442d-a004-96d2c4e73a3b","Type":"ContainerDied","Data":"0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f"}
Nov 02 15:54:19.057945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.057587    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj" event={"ID":"15d6fd17-fdd7-442d-a004-96d2c4e73a3b","Type":"ContainerDied","Data":"271aa8660ba901f534eef6334082d3956cbd4a961013d9c57e48f3b78a4b5573"}
Nov 02 15:54:19.057945 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.057618    5041 scope.go:122] "RemoveContainer" containerID="0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f"
Nov 02 15:54:19.060060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.060023    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"
Nov 02 15:54:19.064699 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.064674    5041 generic.go:356] "Generic (PLEG): container finished" podID="1d0b63d7-ab28-495d-8a45-45379a3891e1" containerID="2941c40016c7f33958953876f046812be5b70f07f1ffbd6e2ca3a3856830b49d" exitCode=0
Nov 02 15:54:19.064809 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.064709    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph" event={"ID":"1d0b63d7-ab28-495d-8a45-45379a3891e1","Type":"ContainerDied","Data":"2941c40016c7f33958953876f046812be5b70f07f1ffbd6e2ca3a3856830b49d"}
Nov 02 15:54:19.074301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.074239    5041 scope.go:122] "RemoveContainer" containerID="0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f"
Nov 02 15:54:19.074641 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:19.074613    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f\": container with ID starting with 0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f not found: ID does not exist" containerID="0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f"
Nov 02 15:54:19.074763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.074653    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f"} err="failed to get container status \"0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f\": rpc error: code = NotFound desc = could not find container \"0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f\": container with ID starting with 0d7a70ab21d87bc142ad9b50deb229146a2543cdde2eebac2bb3887f6aec942f not found: ID does not exist"
Nov 02 15:54:19.125583 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.125551    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs\") pod \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") "
Nov 02 15:54:19.125727 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.125677    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls\") pod \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") "
Nov 02 15:54:19.125803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.125788    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj\" (UniqueName: \"kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj\") pod \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") "
Nov 02 15:54:19.126211 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.126187    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca\") pod \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\" (UID: \"15d6fd17-fdd7-442d-a004-96d2c4e73a3b\") "
Nov 02 15:54:19.126648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.126606    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca" pod "15d6fd17-fdd7-442d-a004-96d2c4e73a3b" (UID: "15d6fd17-fdd7-442d-a004-96d2c4e73a3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:19.126769 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.126750    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.128489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.128420    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs" pod "15d6fd17-fdd7-442d-a004-96d2c4e73a3b" (UID: "15d6fd17-fdd7-442d-a004-96d2c4e73a3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:19.128600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.128489    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj" pod "15d6fd17-fdd7-442d-a004-96d2c4e73a3b" (UID: "15d6fd17-fdd7-442d-a004-96d2c4e73a3b"). InnerVolumeSpecName "kube-api-access-prgwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:19.128600 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.128530    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls" pod "15d6fd17-fdd7-442d-a004-96d2c4e73a3b" (UID: "15d6fd17-fdd7-442d-a004-96d2c4e73a3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:19.140213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.140194    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:54:19.151369 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.151340    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:54:19.228196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.228149    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume\") pod \"1d0b63d7-ab28-495d-8a45-45379a3891e1\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") "
Nov 02 15:54:19.228359 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.228209    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47\" (UniqueName: \"kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47\") pod \"1d0b63d7-ab28-495d-8a45-45379a3891e1\" (UID: \"1d0b63d7-ab28-495d-8a45-45379a3891e1\") "
Nov 02 15:54:19.228673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.228639    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume" pod "1d0b63d7-ab28-495d-8a45-45379a3891e1" (UID: "1d0b63d7-ab28-495d-8a45-45379a3891e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:19.228673 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.228669    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.228843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.228694    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.228843 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.228709    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgwj\" (UniqueName: \"kubernetes.io/projected/15d6fd17-fdd7-442d-a004-96d2c4e73a3b-kube-api-access-prgwj\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.230423 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.230400    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47" pod "1d0b63d7-ab28-495d-8a45-45379a3891e1" (UID: "1d0b63d7-ab28-495d-8a45-45379a3891e1"). InnerVolumeSpecName "kube-api-access-qrl47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:19.329690 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.329665    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz\" (UniqueName: \"kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz\") pod \"c22dfaed-e76a-4b46-b522-9f4a6d119018\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") "
Nov 02 15:54:19.329833 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.329765    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume\") pod \"c22dfaed-e76a-4b46-b522-9f4a6d119018\" (UID: \"c22dfaed-e76a-4b46-b522-9f4a6d119018\") "
Nov 02 15:54:19.329962 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.329949    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0b63d7-ab28-495d-8a45-45379a3891e1-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.330036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.329962    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrl47\" (UniqueName: \"kubernetes.io/projected/1d0b63d7-ab28-495d-8a45-45379a3891e1-kube-api-access-qrl47\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.330174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.330152    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume" pod "c22dfaed-e76a-4b46-b522-9f4a6d119018" (UID: "c22dfaed-e76a-4b46-b522-9f4a6d119018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:19.331626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.331605    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz" pod "c22dfaed-e76a-4b46-b522-9f4a6d119018" (UID: "c22dfaed-e76a-4b46-b522-9f4a6d119018"). InnerVolumeSpecName "kube-api-access-d27pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:19.412312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.412287    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"]
Nov 02 15:54:19.422217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.422192    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8537e9-6f789457b-bsmdj"]
Nov 02 15:54:19.431007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.430982    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22dfaed-e76a-4b46-b522-9f4a6d119018-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.431007 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.431006    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27pz\" (UniqueName: \"kubernetes.io/projected/c22dfaed-e76a-4b46-b522-9f4a6d119018-kube-api-access-d27pz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:19.479385 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.479350    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"]
Nov 02 15:54:19.484691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.484670    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8537e9-tw5kw"]
Nov 02 15:54:19.887797 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.887712    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d6fd17-fdd7-442d-a004-96d2c4e73a3b" path="/var/lib/kubelet/pods/15d6fd17-fdd7-442d-a004-96d2c4e73a3b/volumes"
Nov 02 15:54:19.888157 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:19.888146    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36c184d-bc95-4465-abef-fd5e37271d75" path="/var/lib/kubelet/pods/d36c184d-bc95-4465-abef-fd5e37271d75/volumes"
Nov 02 15:54:20.058230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.058198    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"]
Nov 02 15:54:20.072035 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.072007    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"
Nov 02 15:54:20.072179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.072025    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp" event={"ID":"c22dfaed-e76a-4b46-b522-9f4a6d119018","Type":"ContainerDied","Data":"62fc448f83cc14cf80954235aee636277d2e1d9642fed4bd660a0960eedd3153"}
Nov 02 15:54:20.072179 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.072085    5041 scope.go:122] "RemoveContainer" containerID="7a60564fa42616c65e96552f7b8d651f439ff6c5c3ace90e111b0db7b64e0903"
Nov 02 15:54:20.075457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.075438    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"
Nov 02 15:54:20.075566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.075470    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph" event={"ID":"1d0b63d7-ab28-495d-8a45-45379a3891e1","Type":"ContainerDied","Data":"78cccabadbce04a8062bf65201928e003436a0b38551ae42de568dcebfb54086"}
Nov 02 15:54:20.082731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.082710    5041 scope.go:122] "RemoveContainer" containerID="2941c40016c7f33958953876f046812be5b70f07f1ffbd6e2ca3a3856830b49d"
Nov 02 15:54:20.089310 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.089285    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"]
Nov 02 15:54:20.163094 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.163030    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"]
Nov 02 15:54:20.189425 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.189401    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8537e9-68b97f5df5-pxvph"]
Nov 02 15:54:20.279262 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.279232    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"]
Nov 02 15:54:20.282609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:20.282580    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-5bd5bdc794-h2rzp"]
Nov 02 15:54:21.887022 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:21.886993    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0b63d7-ab28-495d-8a45-45379a3891e1" path="/var/lib/kubelet/pods/1d0b63d7-ab28-495d-8a45-45379a3891e1/volumes"
Nov 02 15:54:21.887408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:21.887393    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22dfaed-e76a-4b46-b522-9f4a6d119018" path="/var/lib/kubelet/pods/c22dfaed-e76a-4b46-b522-9f4a6d119018/volumes"
Nov 02 15:54:22.054976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.054953    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:54:22.089063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.089039    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:54:22.091213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.091189    5041 generic.go:356] "Generic (PLEG): container finished" podID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerID="02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00" exitCode=0
Nov 02 15:54:22.091321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.091227    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"
Nov 02 15:54:22.091321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.091265    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" event={"ID":"e9e68191-fd7d-400a-8a9c-3e6bbc242c81","Type":"ContainerDied","Data":"02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00"}
Nov 02 15:54:22.091321 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.091306    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg" event={"ID":"e9e68191-fd7d-400a-8a9c-3e6bbc242c81","Type":"ContainerDied","Data":"1fa751c109f2f654b31437819bcdc7fca9af921e28b20579ac90bbaada1c1372"}
Nov 02 15:54:22.091475 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.091330    5041 scope.go:122] "RemoveContainer" containerID="02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00"
Nov 02 15:54:22.092643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.092621    5041 generic.go:356] "Generic (PLEG): container finished" podID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerID="8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1" exitCode=0
Nov 02 15:54:22.092801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.092653    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"
Nov 02 15:54:22.092801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.092698    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" event={"ID":"0f3fdefc-b125-4e26-ac80-5721ea18be6c","Type":"ContainerDied","Data":"8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1"}
Nov 02 15:54:22.092801 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.092741    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj" event={"ID":"0f3fdefc-b125-4e26-ac80-5721ea18be6c","Type":"ContainerDied","Data":"174ada3425241529f4dc2bfcac5e97a5890f76058155bde69b48f970c3a86d73"}
Nov 02 15:54:22.100752 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.100733    5041 scope.go:122] "RemoveContainer" containerID="02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00"
Nov 02 15:54:22.101024 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:22.100998    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00\": container with ID starting with 02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00 not found: ID does not exist" containerID="02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00"
Nov 02 15:54:22.101086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.101028    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00"} err="failed to get container status \"02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00\": rpc error: code = NotFound desc = could not find container \"02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00\": container with ID starting with 02c995b188794a08a43cda60be299f0066f08a345f0347b84ab1d68d25e48d00 not found: ID does not exist"
Nov 02 15:54:22.101086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.101043    5041 scope.go:122] "RemoveContainer" containerID="8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1"
Nov 02 15:54:22.109701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.109684    5041 scope.go:122] "RemoveContainer" containerID="8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1"
Nov 02 15:54:22.109942 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:22.109921    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1\": container with ID starting with 8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1 not found: ID does not exist" containerID="8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1"
Nov 02 15:54:22.110005 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.109948    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1"} err="failed to get container status \"8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1\": rpc error: code = NotFound desc = could not find container \"8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1\": container with ID starting with 8d8634e7dcd1fbdaf1d4b840dea32d3712c559bd9aa94fd81dae81cd717596a1 not found: ID does not exist"
Nov 02 15:54:22.154777 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.154755    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume\") pod \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") "
Nov 02 15:54:22.154891 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.154878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59\" (UniqueName: \"kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59\") pod \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\" (UID: \"e9e68191-fd7d-400a-8a9c-3e6bbc242c81\") "
Nov 02 15:54:22.155098 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.155079    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume" pod "e9e68191-fd7d-400a-8a9c-3e6bbc242c81" (UID: "e9e68191-fd7d-400a-8a9c-3e6bbc242c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:22.156846 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.156829    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59" pod "e9e68191-fd7d-400a-8a9c-3e6bbc242c81" (UID: "e9e68191-fd7d-400a-8a9c-3e6bbc242c81"). InnerVolumeSpecName "kube-api-access-bgx59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:22.255976 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.255938    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5\" (UniqueName: \"kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5\") pod \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") "
Nov 02 15:54:22.256145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.255997    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume\") pod \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\" (UID: \"0f3fdefc-b125-4e26-ac80-5721ea18be6c\") "
Nov 02 15:54:22.256395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.256374    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgx59\" (UniqueName: \"kubernetes.io/projected/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-kube-api-access-bgx59\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:22.256395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.256396    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e68191-fd7d-400a-8a9c-3e6bbc242c81-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:22.256566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.256406    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume" pod "0f3fdefc-b125-4e26-ac80-5721ea18be6c" (UID: "0f3fdefc-b125-4e26-ac80-5721ea18be6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:22.258062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.258042    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5" pod "0f3fdefc-b125-4e26-ac80-5721ea18be6c" (UID: "0f3fdefc-b125-4e26-ac80-5721ea18be6c"). InnerVolumeSpecName "kube-api-access-gz6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:22.357156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.357084    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6w5\" (UniqueName: \"kubernetes.io/projected/0f3fdefc-b125-4e26-ac80-5721ea18be6c-kube-api-access-gz6w5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:22.357156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.357113    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3fdefc-b125-4e26-ac80-5721ea18be6c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:22.478844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.478813    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"]
Nov 02 15:54:22.505305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.505276    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-28da58-57764cf7f5-xqwhj"]
Nov 02 15:54:22.553285 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.553249    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"]
Nov 02 15:54:22.562308 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:22.562277    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-28da58-5b577b4bf7-x28mg"]
Nov 02 15:54:23.888375 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:23.888332    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" path="/var/lib/kubelet/pods/0f3fdefc-b125-4e26-ac80-5721ea18be6c/volumes"
Nov 02 15:54:23.888767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:23.888740    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" path="/var/lib/kubelet/pods/e9e68191-fd7d-400a-8a9c-3e6bbc242c81/volumes"
Nov 02 15:54:25.059388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:25.059331    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerName="payload-processing" containerID="cri-o://e26bf3e04032e52dba585ba0e6b760f1d52eb9b29cfd622ec359a1db61c0ad34" gracePeriod=25
Nov 02 15:54:25.090089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:25.090057    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" containerName="payload-pre-processing" containerID="cri-o://03f5c9f252422a52b5a85d0929eb3aaf2ca44e3ce040a59d7bd46845403d8022" gracePeriod=25
Nov 02 15:54:29.058898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.058846    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" containerID="cri-o://29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653" gracePeriod=45
Nov 02 15:54:29.108814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.108783    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main" containerID="cri-o://3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa" gracePeriod=45
Nov 02 15:54:29.387563 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.387540    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:54:29.394880 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.394861    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424210    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs\") pod \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424260    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir\") pod \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424338    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs\") pod \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424388    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq\" (UniqueName: \"kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq\") pod \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424422    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache\") pod \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424449    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm\") pod \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424481    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home\") pod \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\" (UID: \"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424536    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss\" (UniqueName: \"kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss\") pod \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424564    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm\") pod \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424578    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir" pod "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" (UID: "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home\") pod \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424652    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache\") pod \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424698    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir\") pod \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\" (UID: \"d99d03cc-23a7-4c93-ab72-9d1e59372cd5\") "
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424727    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache" pod "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" (UID: "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.424814    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home" pod "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" (UID: "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.425092    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.425119    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.425134    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.425724    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir" pod "d99d03cc-23a7-4c93-ab72-9d1e59372cd5" (UID: "d99d03cc-23a7-4c93-ab72-9d1e59372cd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.425949    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home" pod "d99d03cc-23a7-4c93-ab72-9d1e59372cd5" (UID: "d99d03cc-23a7-4c93-ab72-9d1e59372cd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.427471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.426097    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache" pod "d99d03cc-23a7-4c93-ab72-9d1e59372cd5" (UID: "d99d03cc-23a7-4c93-ab72-9d1e59372cd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.429424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.429346    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm" pod "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" (UID: "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.429607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.429582    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs" pod "d99d03cc-23a7-4c93-ab72-9d1e59372cd5" (UID: "d99d03cc-23a7-4c93-ab72-9d1e59372cd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:29.429733 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.429619    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs" pod "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" (UID: "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:29.430060 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.430037    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss" pod "d99d03cc-23a7-4c93-ab72-9d1e59372cd5" (UID: "d99d03cc-23a7-4c93-ab72-9d1e59372cd5"). InnerVolumeSpecName "kube-api-access-kcqss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:29.430160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.430119    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm" pod "d99d03cc-23a7-4c93-ab72-9d1e59372cd5" (UID: "d99d03cc-23a7-4c93-ab72-9d1e59372cd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.431207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.431185    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq" pod "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" (UID: "5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f"). InnerVolumeSpecName "kube-api-access-flxqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:29.526299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526267    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcqss\" (UniqueName: \"kubernetes.io/projected/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-kube-api-access-kcqss\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526297    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526308    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526315    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526326    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526334    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526342    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99d03cc-23a7-4c93-ab72-9d1e59372cd5-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526350    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flxqq\" (UniqueName: \"kubernetes.io/projected/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-kube-api-access-flxqq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.526543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:29.526374    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:30.144753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.144721    5041 generic.go:356] "Generic (PLEG): container finished" podID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerID="29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653" exitCode=0
Nov 02 15:54:30.145224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.144763    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"
Nov 02 15:54:30.145224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.144808    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" event={"ID":"d99d03cc-23a7-4c93-ab72-9d1e59372cd5","Type":"ContainerDied","Data":"29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653"}
Nov 02 15:54:30.145224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.144854    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" event={"ID":"d99d03cc-23a7-4c93-ab72-9d1e59372cd5","Type":"ContainerDied","Data":"53de9d5714f73c0f14022d27cdafd89a4955f8a8414465ee3fa964e370a595c1"}
Nov 02 15:54:30.145224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.144880    5041 scope.go:122] "RemoveContainer" containerID="29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653"
Nov 02 15:54:30.146442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.146310    5041 generic.go:356] "Generic (PLEG): container finished" podID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerID="3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa" exitCode=0
Nov 02 15:54:30.146442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.146341    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" event={"ID":"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f","Type":"ContainerDied","Data":"3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa"}
Nov 02 15:54:30.146442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.146389    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"
Nov 02 15:54:30.146442 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.146409    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp" event={"ID":"5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f","Type":"ContainerDied","Data":"ada6699ec2836dc2c234870c7cae6d8504b1260dd955676dec37136c20c038f7"}
Nov 02 15:54:30.154746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.154725    5041 scope.go:122] "RemoveContainer" containerID="29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653"
Nov 02 15:54:30.154987 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:30.154966    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653\": container with ID starting with 29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653 not found: ID does not exist" containerID="29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653"
Nov 02 15:54:30.155055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.154994    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653"} err="failed to get container status \"29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653\": rpc error: code = NotFound desc = could not find container \"29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653\": container with ID starting with 29806c0167583eeb86b228d048c8585b94e8f6bebfa8643fbe4f63b74fef0653 not found: ID does not exist"
Nov 02 15:54:30.155055 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.155016    5041 scope.go:122] "RemoveContainer" containerID="3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa"
Nov 02 15:54:30.163975 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.163953    5041 scope.go:122] "RemoveContainer" containerID="3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa"
Nov 02 15:54:30.164248 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:30.164221    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa\": container with ID starting with 3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa not found: ID does not exist" containerID="3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa"
Nov 02 15:54:30.164317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.164257    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa"} err="failed to get container status \"3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa\": rpc error: code = NotFound desc = could not find container \"3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa\": container with ID starting with 3d2deb95b0aae9e145327bcf0ef8edadb94e9c95d3c97830e7c33c74aee498fa not found: ID does not exist"
Nov 02 15:54:30.201595 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.201572    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"]
Nov 02 15:54:30.206241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.206220    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv"]
Nov 02 15:54:30.246808 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.246787    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"]
Nov 02 15:54:30.257289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.257266    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-624254/premium-w3-624254-kserve-6dbfd45d86-79zxp"]
Nov 02 15:54:30.383201 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:30.383172    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-2-w3-624254-kserve-8c69cdcd-6nnnv" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Nov 02 15:54:31.363532 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:31.363484    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:31.363946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:31.363553    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:31.886957 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:31.886920    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" path="/var/lib/kubelet/pods/5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f/volumes"
Nov 02 15:54:31.887444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:31.887419    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" path="/var/lib/kubelet/pods/d99d03cc-23a7-4c93-ab72-9d1e59372cd5/volumes"
Nov 02 15:54:36.768775 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:36.768733    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:54:39.842508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:39.842474    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"]
Nov 02 15:54:39.872647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:39.872616    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"]
Nov 02 15:54:41.363514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:41.363477    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:41.363514 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:41.363492    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:44.843794 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:44.843753    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerName="payload-processing" containerID="cri-o://1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321" gracePeriod=25
Nov 02 15:54:44.873443 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:44.873392    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerName="payload-pre-processing" containerID="cri-o://7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d" gracePeriod=25
Nov 02 15:54:45.170643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.170611    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"]
Nov 02 15:54:45.170863 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.170837    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5" podUID="bd10f188-3b57-4220-a11d-33972f7b069f" containerName="maas-api" containerID="cri-o://cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e" gracePeriod=30
Nov 02 15:54:45.255095 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.255062    5041 generic.go:356] "Generic (PLEG): container finished" podID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerID="e26bf3e04032e52dba585ba0e6b760f1d52eb9b29cfd622ec359a1db61c0ad34" exitCode=0
Nov 02 15:54:45.255406 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.255384    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" event={"ID":"c7b7111b-0097-4bba-84b5-bf0493ee7910","Type":"ContainerDied","Data":"e26bf3e04032e52dba585ba0e6b760f1d52eb9b29cfd622ec359a1db61c0ad34"}
Nov 02 15:54:45.257967 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.257942    5041 generic.go:356] "Generic (PLEG): container finished" podID="e5421600-ded3-4025-9ea9-239c536e94cb" containerID="03f5c9f252422a52b5a85d0929eb3aaf2ca44e3ce040a59d7bd46845403d8022" exitCode=0
Nov 02 15:54:45.258222 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.258202    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" event={"ID":"e5421600-ded3-4025-9ea9-239c536e94cb","Type":"ContainerDied","Data":"03f5c9f252422a52b5a85d0929eb3aaf2ca44e3ce040a59d7bd46845403d8022"}
Nov 02 15:54:45.355353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.355320    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:54:45.419203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.419165    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:45.442897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.442864    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:54:45.470084 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.470058    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6\" (UniqueName: \"kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6\") pod \"c7b7111b-0097-4bba-84b5-bf0493ee7910\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") "
Nov 02 15:54:45.470237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.470114    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume\") pod \"c7b7111b-0097-4bba-84b5-bf0493ee7910\" (UID: \"c7b7111b-0097-4bba-84b5-bf0493ee7910\") "
Nov 02 15:54:45.470237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.470154    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt\" (UniqueName: \"kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt\") pod \"e5421600-ded3-4025-9ea9-239c536e94cb\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") "
Nov 02 15:54:45.470237 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.470172    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume\") pod \"e5421600-ded3-4025-9ea9-239c536e94cb\" (UID: \"e5421600-ded3-4025-9ea9-239c536e94cb\") "
Nov 02 15:54:45.470571 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.470549    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume" pod "c7b7111b-0097-4bba-84b5-bf0493ee7910" (UID: "c7b7111b-0097-4bba-84b5-bf0493ee7910"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:45.470656 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.470634    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume" pod "e5421600-ded3-4025-9ea9-239c536e94cb" (UID: "e5421600-ded3-4025-9ea9-239c536e94cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:45.472740 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.472716    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt" pod "e5421600-ded3-4025-9ea9-239c536e94cb" (UID: "e5421600-ded3-4025-9ea9-239c536e94cb"). InnerVolumeSpecName "kube-api-access-2czlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:45.472844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.472747    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6" pod "c7b7111b-0097-4bba-84b5-bf0493ee7910" (UID: "c7b7111b-0097-4bba-84b5-bf0493ee7910"). InnerVolumeSpecName "kube-api-access-nt6l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:45.485036 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.485014    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:54:45.572288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572255    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls\") pod \"bd10f188-3b57-4220-a11d-33972f7b069f\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") "
Nov 02 15:54:45.572288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572298    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs\") pod \"bd10f188-3b57-4220-a11d-33972f7b069f\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") "
Nov 02 15:54:45.572576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572375    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca\") pod \"bd10f188-3b57-4220-a11d-33972f7b069f\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") "
Nov 02 15:54:45.572576 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572433    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt\" (UniqueName: \"kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt\") pod \"bd10f188-3b57-4220-a11d-33972f7b069f\" (UID: \"bd10f188-3b57-4220-a11d-33972f7b069f\") "
Nov 02 15:54:45.572802 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572761    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca" pod "bd10f188-3b57-4220-a11d-33972f7b069f" (UID: "bd10f188-3b57-4220-a11d-33972f7b069f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:45.572953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572806    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6l6\" (UniqueName: \"kubernetes.io/projected/c7b7111b-0097-4bba-84b5-bf0493ee7910-kube-api-access-nt6l6\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.572953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572831    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b7111b-0097-4bba-84b5-bf0493ee7910-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.572953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572845    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czlt\" (UniqueName: \"kubernetes.io/projected/e5421600-ded3-4025-9ea9-239c536e94cb-kube-api-access-2czlt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.572953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.572858    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5421600-ded3-4025-9ea9-239c536e94cb-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.574554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.574527    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt" pod "bd10f188-3b57-4220-a11d-33972f7b069f" (UID: "bd10f188-3b57-4220-a11d-33972f7b069f"). InnerVolumeSpecName "kube-api-access-z5wgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:45.574648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.574561    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs" pod "bd10f188-3b57-4220-a11d-33972f7b069f" (UID: "bd10f188-3b57-4220-a11d-33972f7b069f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:45.574648 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.574592    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls" pod "bd10f188-3b57-4220-a11d-33972f7b069f" (UID: "bd10f188-3b57-4220-a11d-33972f7b069f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:45.673564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.673539    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.673564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.673563    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10f188-3b57-4220-a11d-33972f7b069f-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.673734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.673573    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd10f188-3b57-4220-a11d-33972f7b069f-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:45.673734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:45.673583    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wgt\" (UniqueName: \"kubernetes.io/projected/bd10f188-3b57-4220-a11d-33972f7b069f-kube-api-access-z5wgt\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:54:46.269793 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.269754    5041 generic.go:356] "Generic (PLEG): container finished" podID="bd10f188-3b57-4220-a11d-33972f7b069f" containerID="cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e" exitCode=0
Nov 02 15:54:46.270274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.269803    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"
Nov 02 15:54:46.270274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.269845    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5" event={"ID":"bd10f188-3b57-4220-a11d-33972f7b069f","Type":"ContainerDied","Data":"cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e"}
Nov 02 15:54:46.270274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.269911    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5" event={"ID":"bd10f188-3b57-4220-a11d-33972f7b069f","Type":"ContainerDied","Data":"9e1e925b01dedc153b8a5fab40873b84aae6a69a4bd7d205ba73883c3af296a0"}
Nov 02 15:54:46.270274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.269934    5041 scope.go:122] "RemoveContainer" containerID="cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e"
Nov 02 15:54:46.271662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.271530    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"
Nov 02 15:54:46.271662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.271561    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6" event={"ID":"e5421600-ded3-4025-9ea9-239c536e94cb","Type":"ContainerDied","Data":"e96a66d92242743fd1eab3fe09268d0ed43418f4b531c0dc8ad54560ff00f81f"}
Nov 02 15:54:46.273438 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.273410    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt" event={"ID":"c7b7111b-0097-4bba-84b5-bf0493ee7910","Type":"ContainerDied","Data":"8f92f59af63908ad25f2cd9fc3cccce89d8b3943e9376c52f2ad06aba6dccc42"}
Nov 02 15:54:46.273527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.273473    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"
Nov 02 15:54:46.285286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.283978    5041 scope.go:122] "RemoveContainer" containerID="cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e"
Nov 02 15:54:46.285286 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:54:46.284426    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e\": container with ID starting with cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e not found: ID does not exist" containerID="cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e"
Nov 02 15:54:46.285286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.284459    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e"} err="failed to get container status \"cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e\": rpc error: code = NotFound desc = could not find container \"cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e\": container with ID starting with cba7d55f616cb50a8447cb750192745b99aab9b79527fad31dced1c95f3a896e not found: ID does not exist"
Nov 02 15:54:46.285286 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.284483    5041 scope.go:122] "RemoveContainer" containerID="03f5c9f252422a52b5a85d0929eb3aaf2ca44e3ce040a59d7bd46845403d8022"
Nov 02 15:54:46.296634 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.296613    5041 scope.go:122] "RemoveContainer" containerID="e26bf3e04032e52dba585ba0e6b760f1d52eb9b29cfd622ec359a1db61c0ad34"
Nov 02 15:54:46.344626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.343663    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"]
Nov 02 15:54:46.352934 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.352904    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-624254-557bb7f4c5-xggrt"]
Nov 02 15:54:46.386092 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.386064    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"]
Nov 02 15:54:46.399971 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.398329    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-624254-66d774b675-lzdv6"]
Nov 02 15:54:46.431232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.431201    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"]
Nov 02 15:54:46.451230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:46.451200    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fda646-864497c546-296k5"]
Nov 02 15:54:47.887080 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:47.887050    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd10f188-3b57-4220-a11d-33972f7b069f" path="/var/lib/kubelet/pods/bd10f188-3b57-4220-a11d-33972f7b069f/volumes"
Nov 02 15:54:47.887473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:47.887440    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" path="/var/lib/kubelet/pods/c7b7111b-0097-4bba-84b5-bf0493ee7910/volumes"
Nov 02 15:54:47.887787 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:47.887773    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" path="/var/lib/kubelet/pods/e5421600-ded3-4025-9ea9-239c536e94cb/volumes"
Nov 02 15:54:54.373611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:54.373565    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:54.611735 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:54.611694    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"]
Nov 02 15:54:54.611959 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:54.611916    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy" containerID="cri-o://c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38" gracePeriod=30
Nov 02 15:54:55.399847 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:55.399807    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:54:59.875294 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:54:59.875269    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:55:00.011663 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011616    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.011874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011681    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.011874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011714    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.011874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011744    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.011874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011784    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.011874 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011850    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.012181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011918    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.012181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011953    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:00.012181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.011988    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.012181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012037    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy\") pod \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\" (UID: \"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0\") "
Nov 02 15:55:00.012181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012042    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:00.012181 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012159    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:00.012556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012335    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:00.012556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012408    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:00.012556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012490    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.012556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012511    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.012556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012526    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.012556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.012543    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.014193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.014169    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:00.014404 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.014382    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "kube-api-access-r5d7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:00.014662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.014641    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:55:00.014715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.014654    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy" pod "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" (UID: "d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:00.113450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.113422    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.113450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.113446    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.113450 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.113457    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.113653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.113465    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5d7c\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-kube-api-access-r5d7c\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.113653 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.113475    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:00.362593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.362517    5041 generic.go:356] "Generic (PLEG): container finished" podID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerID="c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38" exitCode=0
Nov 02 15:55:00.362593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.362569    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"
Nov 02 15:55:00.362910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.362604    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" event={"ID":"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0","Type":"ContainerDied","Data":"c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38"}
Nov 02 15:55:00.362910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.362659    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x" event={"ID":"d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0","Type":"ContainerDied","Data":"ef46db6ba112d2a7c478857d13fa513f6114e5cd3100b9732a914859aeb2266c"}
Nov 02 15:55:00.362910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.362691    5041 scope.go:122] "RemoveContainer" containerID="c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38"
Nov 02 15:55:00.372895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.372876    5041 scope.go:122] "RemoveContainer" containerID="dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"
Nov 02 15:55:00.381566 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.381547    5041 scope.go:122] "RemoveContainer" containerID="c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38"
Nov 02 15:55:00.381821 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:00.381799    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38\": container with ID starting with c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38 not found: ID does not exist" containerID="c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38"
Nov 02 15:55:00.381872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.381830    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38"} err="failed to get container status \"c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38\": rpc error: code = NotFound desc = could not find container \"c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38\": container with ID starting with c2fe34cbc7213270cefdd05223d05e3bdb4d830830f2f4f8c7f8e433b05e2c38 not found: ID does not exist"
Nov 02 15:55:00.381872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.381847    5041 scope.go:122] "RemoveContainer" containerID="dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"
Nov 02 15:55:00.382092 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:00.382070    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d\": container with ID starting with dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d not found: ID does not exist" containerID="dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"
Nov 02 15:55:00.382135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.382098    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d"} err="failed to get container status \"dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d\": rpc error: code = NotFound desc = could not find container \"dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d\": container with ID starting with dcd96aa25a4f1b388064b181e8ccd86d13ea9de0f0bbdf71b120a5377783433d not found: ID does not exist"
Nov 02 15:55:00.417613 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.417588    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"]
Nov 02 15:55:00.420935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:00.420913    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-csm5x"]
Nov 02 15:55:01.886702 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:01.886670    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" path="/var/lib/kubelet/pods/d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0/volumes"
Nov 02 15:55:04.372947 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:04.372909    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:05.130020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.129996    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:55:05.149785 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.149763    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:55:05.257579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.257542    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc\" (UniqueName: \"kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc\") pod \"502b3812-a75a-46ca-b156-2189ef4c8d29\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") "
Nov 02 15:55:05.257730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.257646    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2\" (UniqueName: \"kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2\") pod \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") "
Nov 02 15:55:05.257730 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.257672    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume\") pod \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\" (UID: \"a61a32ad-3972-472b-807a-a9ec6a6bcd6b\") "
Nov 02 15:55:05.257817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.257781    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume\") pod \"502b3812-a75a-46ca-b156-2189ef4c8d29\" (UID: \"502b3812-a75a-46ca-b156-2189ef4c8d29\") "
Nov 02 15:55:05.258180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.258147    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume" pod "a61a32ad-3972-472b-807a-a9ec6a6bcd6b" (UID: "a61a32ad-3972-472b-807a-a9ec6a6bcd6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:05.258337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.258189    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume" pod "502b3812-a75a-46ca-b156-2189ef4c8d29" (UID: "502b3812-a75a-46ca-b156-2189ef4c8d29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:05.259685 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.259663    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc" pod "502b3812-a75a-46ca-b156-2189ef4c8d29" (UID: "502b3812-a75a-46ca-b156-2189ef4c8d29"). InnerVolumeSpecName "kube-api-access-dj7pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:05.259772 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.259735    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2" pod "a61a32ad-3972-472b-807a-a9ec6a6bcd6b" (UID: "a61a32ad-3972-472b-807a-a9ec6a6bcd6b"). InnerVolumeSpecName "kube-api-access-tmkp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:05.358889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.358833    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502b3812-a75a-46ca-b156-2189ef4c8d29-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:05.358889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.358856    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7pc\" (UniqueName: \"kubernetes.io/projected/502b3812-a75a-46ca-b156-2189ef4c8d29-kube-api-access-dj7pc\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:05.358889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.358866    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkp2\" (UniqueName: \"kubernetes.io/projected/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-kube-api-access-tmkp2\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:05.358889 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.358877    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61a32ad-3972-472b-807a-a9ec6a6bcd6b-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:05.392714 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.392689    5041 generic.go:356] "Generic (PLEG): container finished" podID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerID="1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321" exitCode=0
Nov 02 15:55:05.393066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.392732    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"
Nov 02 15:55:05.393066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.392779    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" event={"ID":"a61a32ad-3972-472b-807a-a9ec6a6bcd6b","Type":"ContainerDied","Data":"1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321"}
Nov 02 15:55:05.393066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.392828    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm" event={"ID":"a61a32ad-3972-472b-807a-a9ec6a6bcd6b","Type":"ContainerDied","Data":"2e04e09312c62b8b014721874ba9595c9a1ce79423764bbaa9c1fbe1286280c6"}
Nov 02 15:55:05.393066 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.392857    5041 scope.go:122] "RemoveContainer" containerID="1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321"
Nov 02 15:55:05.394241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.394217    5041 generic.go:356] "Generic (PLEG): container finished" podID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerID="7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d" exitCode=0
Nov 02 15:55:05.394335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.394268    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" event={"ID":"502b3812-a75a-46ca-b156-2189ef4c8d29","Type":"ContainerDied","Data":"7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d"}
Nov 02 15:55:05.394335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.394276    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"
Nov 02 15:55:05.394335 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.394305    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn" event={"ID":"502b3812-a75a-46ca-b156-2189ef4c8d29","Type":"ContainerDied","Data":"7aa0345349775e85e3226a4fdeb1dc898c576a9b27e8109cf9e3184286f03754"}
Nov 02 15:55:05.402426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.402405    5041 scope.go:122] "RemoveContainer" containerID="1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321"
Nov 02 15:55:05.402740 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:05.402718    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321\": container with ID starting with 1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321 not found: ID does not exist" containerID="1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321"
Nov 02 15:55:05.402806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.402745    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321"} err="failed to get container status \"1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321\": rpc error: code = NotFound desc = could not find container \"1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321\": container with ID starting with 1b1e941c27870b9ef2da1bcb51520e802ab21daa00267825597a9cf5fe970321 not found: ID does not exist"
Nov 02 15:55:05.402806 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.402763    5041 scope.go:122] "RemoveContainer" containerID="7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d"
Nov 02 15:55:05.411452 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.411434    5041 scope.go:122] "RemoveContainer" containerID="7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d"
Nov 02 15:55:05.411713 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:05.411686    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d\": container with ID starting with 7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d not found: ID does not exist" containerID="7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d"
Nov 02 15:55:05.411783 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.411722    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d"} err="failed to get container status \"7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d\": rpc error: code = NotFound desc = could not find container \"7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d\": container with ID starting with 7e63c275009b0da6d5d4422054e637013758b64e0e744c0bc400cfba31092d6d not found: ID does not exist"
Nov 02 15:55:05.447822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.447798    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"]
Nov 02 15:55:05.452410 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.452356    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fda646-7645f6dcb5-frtnm"]
Nov 02 15:55:05.485828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.485809    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"]
Nov 02 15:55:05.489914 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.489893    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fda646-5f977fc867-ccvvn"]
Nov 02 15:55:05.708229 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.708205    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"]
Nov 02 15:55:05.743768 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.743743    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"]
Nov 02 15:55:05.890293 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.890258    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" path="/var/lib/kubelet/pods/502b3812-a75a-46ca-b156-2189ef4c8d29/volumes"
Nov 02 15:55:05.890712 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:05.890696    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" path="/var/lib/kubelet/pods/a61a32ad-3972-472b-807a-a9ec6a6bcd6b/volumes"
Nov 02 15:55:06.768784 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:06.768748    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:55:08.016836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.016799    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"]
Nov 02 15:55:08.017272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.016978    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" podUID="79b25021-d58b-46c7-95a0-5f13c7ceb327" containerName="maas-api" containerID="cri-o://a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600" gracePeriod=30
Nov 02 15:55:08.279720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.279653    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:55:08.387500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.387471    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\") pod \"79b25021-d58b-46c7-95a0-5f13c7ceb327\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") "
Nov 02 15:55:08.387679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.387547    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca\") pod \"79b25021-d58b-46c7-95a0-5f13c7ceb327\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") "
Nov 02 15:55:08.387679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.387602    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs\") pod \"79b25021-d58b-46c7-95a0-5f13c7ceb327\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") "
Nov 02 15:55:08.387679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.387674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s\" (UniqueName: \"kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s\") pod \"79b25021-d58b-46c7-95a0-5f13c7ceb327\" (UID: \"79b25021-d58b-46c7-95a0-5f13c7ceb327\") "
Nov 02 15:55:08.387990 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.387949    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca" pod "79b25021-d58b-46c7-95a0-5f13c7ceb327" (UID: "79b25021-d58b-46c7-95a0-5f13c7ceb327"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:08.389856 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.389827    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls" pod "79b25021-d58b-46c7-95a0-5f13c7ceb327" (UID: "79b25021-d58b-46c7-95a0-5f13c7ceb327"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:08.389958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.389861    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs" pod "79b25021-d58b-46c7-95a0-5f13c7ceb327" (UID: "79b25021-d58b-46c7-95a0-5f13c7ceb327"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:08.390105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.390085    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s" pod "79b25021-d58b-46c7-95a0-5f13c7ceb327" (UID: "79b25021-d58b-46c7-95a0-5f13c7ceb327"). InnerVolumeSpecName "kube-api-access-zrt6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:08.414585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.414559    5041 generic.go:356] "Generic (PLEG): container finished" podID="79b25021-d58b-46c7-95a0-5f13c7ceb327" containerID="a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600" exitCode=0
Nov 02 15:55:08.414683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.414606    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"
Nov 02 15:55:08.414683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.414633    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" event={"ID":"79b25021-d58b-46c7-95a0-5f13c7ceb327","Type":"ContainerDied","Data":"a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600"}
Nov 02 15:55:08.414683 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.414682    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6" event={"ID":"79b25021-d58b-46c7-95a0-5f13c7ceb327","Type":"ContainerDied","Data":"bdfcb4d0e45062bd3d8d9507328d2f97d25e7acb4e7788c6bd150f5d54102b3f"}
Nov 02 15:55:08.414828 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.414706    5041 scope.go:122] "RemoveContainer" containerID="a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600"
Nov 02 15:55:08.424763 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.424741    5041 scope.go:122] "RemoveContainer" containerID="a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600"
Nov 02 15:55:08.425024 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:08.425004    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600\": container with ID starting with a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600 not found: ID does not exist" containerID="a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600"
Nov 02 15:55:08.425077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.425034    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600"} err="failed to get container status \"a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600\": rpc error: code = NotFound desc = could not find container \"a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600\": container with ID starting with a1d0345198e00cffc4f7c72d2323028ced5842ec1b4c35b02945b6398c4f1600 not found: ID does not exist"
Nov 02 15:55:08.468537 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.468510    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"]
Nov 02 15:55:08.470825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.470799    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce791-79b45cb96d-84jp6"]
Nov 02 15:55:08.488593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.488568    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrt6s\" (UniqueName: \"kubernetes.io/projected/79b25021-d58b-46c7-95a0-5f13c7ceb327-kube-api-access-zrt6s\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:08.488695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.488595    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:08.488695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.488610    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25021-d58b-46c7-95a0-5f13c7ceb327-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:08.488695 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:08.488623    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25021-d58b-46c7-95a0-5f13c7ceb327-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:09.886814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:09.886781    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b25021-d58b-46c7-95a0-5f13c7ceb327" path="/var/lib/kubelet/pods/79b25021-d58b-46c7-95a0-5f13c7ceb327/volumes"
Nov 02 15:55:10.708519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:10.708455    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerName="payload-processing" containerID="cri-o://a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad" gracePeriod=25
Nov 02 15:55:10.745108 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:10.745072    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerName="payload-pre-processing" containerID="cri-o://dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483" gracePeriod=25
Nov 02 15:55:15.043622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:15.043525    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:15.044814 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:15.044781    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:17.317148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317106    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"]
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317625    5041 cpu_manager.go:414] "containerMap: removing container" podUID="79b25021-d58b-46c7-95a0-5f13c7ceb327" containerName="maas-api"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317641    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b25021-d58b-46c7-95a0-5f13c7ceb327" containerName="maas-api"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317654    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d36c184d-bc95-4465-abef-fd5e37271d75" containerName="revoke-keys"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317659    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36c184d-bc95-4465-abef-fd5e37271d75" containerName="revoke-keys"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317669    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bd10f188-3b57-4220-a11d-33972f7b069f" containerName="maas-api"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317674    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd10f188-3b57-4220-a11d-33972f7b069f" containerName="maas-api"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317682    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy"
Nov 02 15:55:17.317680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317687    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317698    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317705    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317712    5041 cpu_manager.go:414] "containerMap: removing container" podUID="15d6fd17-fdd7-442d-a004-96d2c4e73a3b" containerName="maas-api"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317717    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d6fd17-fdd7-442d-a004-96d2c4e73a3b" containerName="maas-api"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317726    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317731    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317738    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317742    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317751    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317755    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317763    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317768    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317776    5041 cpu_manager.go:414] "containerMap: removing container" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317781    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317791    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317795    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317808    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317812    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317819    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c22dfaed-e76a-4b46-b522-9f4a6d119018" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317823    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22dfaed-e76a-4b46-b522-9f4a6d119018" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317832    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317836    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317843    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317847    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317855    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1d0b63d7-ab28-495d-8a45-45379a3891e1" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317859    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0b63d7-ab28-495d-8a45-45379a3891e1" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317867    5041 cpu_manager.go:414] "containerMap: removing container" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317871    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317936    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b7111b-0097-4bba-84b5-bf0493ee7910" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317943    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d6fd17-fdd7-442d-a004-96d2c4e73a3b" containerName="maas-api"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317949    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3fdefc-b125-4e26-ac80-5721ea18be6c" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317955    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e68191-fd7d-400a-8a9c-3e6bbc242c81" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317961    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317967    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa3eea8-a2a6-4608-b93e-dfce5eec5034" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317974    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4307bc-148b-44b3-a773-a137aad7ca1b" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317980    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5421600-ded3-4025-9ea9-239c536e94cb" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317986    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a83c0e2-0bc1-4e1b-aead-6dab74b2e66f" containerName="main"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317991    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22dfaed-e76a-4b46-b522-9f4a6d119018" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.317997    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99d03cc-23a7-4c93-ab72-9d1e59372cd5" containerName="main"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.318002    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61a32ad-3972-472b-807a-a9ec6a6bcd6b" containerName="payload-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.318007    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0b63d7-ab28-495d-8a45-45379a3891e1" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.318013    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="502b3812-a75a-46ca-b156-2189ef4c8d29" containerName="payload-pre-processing"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.318019    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b25021-d58b-46c7-95a0-5f13c7ceb327" containerName="maas-api"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.318025    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36c184d-bc95-4465-abef-fd5e37271d75" containerName="revoke-keys"
Nov 02 15:55:17.318068 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.318030    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd10f188-3b57-4220-a11d-33972f7b069f" containerName="maas-api"
Nov 02 15:55:17.322677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.322656    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.350190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.350143    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:55:17.350302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.350164    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:55:17.353116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.353090    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"]
Nov 02 15:55:17.388087 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.388066    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:55:17.388204 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.388145    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:55:17.470964 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.470931    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.471106 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.470998    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.471184 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.471106    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.571701 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.571633    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.571824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.571790    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.571876 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.571830    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.572492 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.572463    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.574062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.574043    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.594933 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.594910    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67\") pod \"maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.636489 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.636464    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:17.937871 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:17.937844    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"]
Nov 02 15:55:17.939533 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:55:17.939501    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb977e400_c791_4b0f_a753_7ae2094d2742.slice/crio-7554dcaf4338976d056775b5e6dc1913976e7128e9d8b358f58e6734cbf89313 WatchSource:0}: Error finding container 7554dcaf4338976d056775b5e6dc1913976e7128e9d8b358f58e6734cbf89313: Status 404 returned error can't find the container with id 7554dcaf4338976d056775b5e6dc1913976e7128e9d8b358f58e6734cbf89313
Nov 02 15:55:18.473568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:18.473482    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t" event={"ID":"b977e400-c791-4b0f-a753-7ae2094d2742","Type":"ContainerStarted","Data":"f16336d4697c249a4d14e9e48565d7a88a5b2791fe73f724d66bd4b24f5516e0"}
Nov 02 15:55:18.473568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:18.473545    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t" event={"ID":"b977e400-c791-4b0f-a753-7ae2094d2742","Type":"ContainerStarted","Data":"7554dcaf4338976d056775b5e6dc1913976e7128e9d8b358f58e6734cbf89313"}
Nov 02 15:55:18.514251 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:18.514197    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t" podStartSLOduration=1.514184027 podStartE2EDuration="1.514184027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:55:18.51214851 +0000 UTC m=+3570.770217960" watchObservedRunningTime="2026-11-02 15:55:18.514184027 +0000 UTC m=+3570.772253477"
Nov 02 15:55:19.481175 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:19.481138    5041 generic.go:356] "Generic (PLEG): container finished" podID="b977e400-c791-4b0f-a753-7ae2094d2742" containerID="f16336d4697c249a4d14e9e48565d7a88a5b2791fe73f724d66bd4b24f5516e0" exitCode=0
Nov 02 15:55:19.481578 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:19.481218    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t" event={"ID":"b977e400-c791-4b0f-a753-7ae2094d2742","Type":"ContainerDied","Data":"f16336d4697c249a4d14e9e48565d7a88a5b2791fe73f724d66bd4b24f5516e0"}
Nov 02 15:55:20.625840 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.625816    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:20.697813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.697783    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67\") pod \"b977e400-c791-4b0f-a753-7ae2094d2742\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") "
Nov 02 15:55:20.697955 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.697840    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca\") pod \"b977e400-c791-4b0f-a753-7ae2094d2742\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") "
Nov 02 15:55:20.699936 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.699906    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67" pod "b977e400-c791-4b0f-a753-7ae2094d2742" (UID: "b977e400-c791-4b0f-a753-7ae2094d2742"). InnerVolumeSpecName "kube-api-access-9hh67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:20.700073 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.699910    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca" pod "b977e400-c791-4b0f-a753-7ae2094d2742" (UID: "b977e400-c791-4b0f-a753-7ae2094d2742"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:20.798617 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.798555    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca\") pod \"b977e400-c791-4b0f-a753-7ae2094d2742\" (UID: \"b977e400-c791-4b0f-a753-7ae2094d2742\") "
Nov 02 15:55:20.798827 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.798814    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/b977e400-c791-4b0f-a753-7ae2094d2742-kube-api-access-9hh67\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:20.798872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.798829    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:20.798872 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.798857    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca" pod "b977e400-c791-4b0f-a753-7ae2094d2742" (UID: "b977e400-c791-4b0f-a753-7ae2094d2742"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:20.899250 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:20.899218    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b977e400-c791-4b0f-a753-7ae2094d2742-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.493819 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:21.493785    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"
Nov 02 15:55:21.493982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:21.493811    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t" event={"ID":"b977e400-c791-4b0f-a753-7ae2094d2742","Type":"ContainerDied","Data":"7554dcaf4338976d056775b5e6dc1913976e7128e9d8b358f58e6734cbf89313"}
Nov 02 15:55:21.493982 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:21.493869    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7554dcaf4338976d056775b5e6dc1913976e7128e9d8b358f58e6734cbf89313"
Nov 02 15:55:22.027753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:22.027710    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"]
Nov 02 15:55:25.042760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:25.042720    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:25.044657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:25.044623    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:27.869302 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:27.869263    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"]
Nov 02 15:55:27.905555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:27.905521    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"]
Nov 02 15:55:28.164836 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.164762    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"]
Nov 02 15:55:28.165205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.165192    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b977e400-c791-4b0f-a753-7ae2094d2742" containerName="revoke-keys"
Nov 02 15:55:28.165263 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.165207    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b977e400-c791-4b0f-a753-7ae2094d2742" containerName="revoke-keys"
Nov 02 15:55:28.165303 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.165292    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b977e400-c791-4b0f-a753-7ae2094d2742" containerName="revoke-keys"
Nov 02 15:55:28.165353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.165302    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b2c4b5-e9c2-4e7d-95f3-a2240f54eeb0" containerName="istio-proxy"
Nov 02 15:55:28.168591 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.168559    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.189150 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.189127    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:55:28.189295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.189262    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:55:28.189295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.189268    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:55:28.189295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.189289    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:55:28.191191 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.191166    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"]
Nov 02 15:55:28.256444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.256409    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.256588 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.256464    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxz4\" (UniqueName: \"kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.256680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.256650    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.357761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.357729    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.357901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.357776    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxz4\" (UniqueName: \"kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.357901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.357828    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.358472 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.358451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.360065 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.360042    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.380953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.380930    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxz4\" (UniqueName: \"kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4\") pod \"maas-api-revoke-sub-8bfde0b7-nzbrf\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.482603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.482578    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:28.781336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:28.781310    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"]
Nov 02 15:55:28.784212 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:55:28.784182    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f01110d_87a9_467f_95be_4c4195a78793.slice/crio-f16f4de71913c7b7a22968820ce9c7e84188f1ac03e05e1632aa02c609b3c519 WatchSource:0}: Error finding container f16f4de71913c7b7a22968820ce9c7e84188f1ac03e05e1632aa02c609b3c519: Status 404 returned error can't find the container with id f16f4de71913c7b7a22968820ce9c7e84188f1ac03e05e1632aa02c609b3c519
Nov 02 15:55:29.543437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:29.543400    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf" event={"ID":"5f01110d-87a9-467f-95be-4c4195a78793","Type":"ContainerStarted","Data":"de35982f08b53e8a0a4186dd61e000f7f41b1a5bcb866560ee19d98fed7e0117"}
Nov 02 15:55:29.543853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:29.543459    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf" event={"ID":"5f01110d-87a9-467f-95be-4c4195a78793","Type":"ContainerStarted","Data":"f16f4de71913c7b7a22968820ce9c7e84188f1ac03e05e1632aa02c609b3c519"}
Nov 02 15:55:29.585316 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:29.585256    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf" podStartSLOduration=1.585238572 podStartE2EDuration="1.585238572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:55:29.582747551 +0000 UTC m=+3581.840817001" watchObservedRunningTime="2026-11-02 15:55:29.585238572 +0000 UTC m=+3581.843308023"
Nov 02 15:55:30.550994 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:30.550958    5041 generic.go:356] "Generic (PLEG): container finished" podID="5f01110d-87a9-467f-95be-4c4195a78793" containerID="de35982f08b53e8a0a4186dd61e000f7f41b1a5bcb866560ee19d98fed7e0117" exitCode=0
Nov 02 15:55:30.551432 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:30.551042    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf" event={"ID":"5f01110d-87a9-467f-95be-4c4195a78793","Type":"ContainerDied","Data":"de35982f08b53e8a0a4186dd61e000f7f41b1a5bcb866560ee19d98fed7e0117"}
Nov 02 15:55:30.991635 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:30.991609    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:55:31.012119 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.012098    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:55:31.079124 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.079099    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume\") pod \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") "
Nov 02 15:55:31.079313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.079214    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume\") pod \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") "
Nov 02 15:55:31.079313 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.079264    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n\" (UniqueName: \"kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n\") pod \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\" (UID: \"50a1a0fb-bea2-4061-b32c-67ed9afeb526\") "
Nov 02 15:55:31.079451 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.079312    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z\" (UniqueName: \"kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z\") pod \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\" (UID: \"fd6539ec-29c7-41d1-ba11-0d1a920530ae\") "
Nov 02 15:55:31.079554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.079533    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume" pod "50a1a0fb-bea2-4061-b32c-67ed9afeb526" (UID: "50a1a0fb-bea2-4061-b32c-67ed9afeb526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:31.079731 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.079704    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a1a0fb-bea2-4061-b32c-67ed9afeb526-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.080145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.080113    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume" pod "fd6539ec-29c7-41d1-ba11-0d1a920530ae" (UID: "fd6539ec-29c7-41d1-ba11-0d1a920530ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:31.081737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.081709    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z" pod "fd6539ec-29c7-41d1-ba11-0d1a920530ae" (UID: "fd6539ec-29c7-41d1-ba11-0d1a920530ae"). InnerVolumeSpecName "kube-api-access-4fh9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:31.082142 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.082116    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n" pod "50a1a0fb-bea2-4061-b32c-67ed9afeb526" (UID: "50a1a0fb-bea2-4061-b32c-67ed9afeb526"). InnerVolumeSpecName "kube-api-access-g958n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:31.180377 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.180337    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fh9z\" (UniqueName: \"kubernetes.io/projected/fd6539ec-29c7-41d1-ba11-0d1a920530ae-kube-api-access-4fh9z\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.180553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.180383    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6539ec-29c7-41d1-ba11-0d1a920530ae-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.180553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.180394    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g958n\" (UniqueName: \"kubernetes.io/projected/50a1a0fb-bea2-4061-b32c-67ed9afeb526-kube-api-access-g958n\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.557746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.557665    5041 generic.go:356] "Generic (PLEG): container finished" podID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerID="dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483" exitCode=0
Nov 02 15:55:31.557746 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.557718    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"
Nov 02 15:55:31.558217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.557749    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" event={"ID":"fd6539ec-29c7-41d1-ba11-0d1a920530ae","Type":"ContainerDied","Data":"dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483"}
Nov 02 15:55:31.558217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.557831    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj" event={"ID":"fd6539ec-29c7-41d1-ba11-0d1a920530ae","Type":"ContainerDied","Data":"2f3da8b09941a5300dd6916de544af632288209323b3e728ecbd34f0572b47a5"}
Nov 02 15:55:31.558217 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.557854    5041 scope.go:122] "RemoveContainer" containerID="dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483"
Nov 02 15:55:31.559500 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.559480    5041 generic.go:356] "Generic (PLEG): container finished" podID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerID="a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad" exitCode=0
Nov 02 15:55:31.559603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.559552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" event={"ID":"50a1a0fb-bea2-4061-b32c-67ed9afeb526","Type":"ContainerDied","Data":"a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad"}
Nov 02 15:55:31.559603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.559576    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"
Nov 02 15:55:31.559603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.559600    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb" event={"ID":"50a1a0fb-bea2-4061-b32c-67ed9afeb526","Type":"ContainerDied","Data":"a3be20514ed31e8c3813e6bf9e59dba2c831212f6c34a97966f9ba171e1fb6cf"}
Nov 02 15:55:31.572815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.572795    5041 scope.go:122] "RemoveContainer" containerID="dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483"
Nov 02 15:55:31.573544 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:31.573519    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483\": container with ID starting with dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483 not found: ID does not exist" containerID="dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483"
Nov 02 15:55:31.573637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.573552    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483"} err="failed to get container status \"dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483\": rpc error: code = NotFound desc = could not find container \"dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483\": container with ID starting with dc7d6d56ae4800fa2229767bb817f5c497489f56c9f33a688553c9ee84ca3483 not found: ID does not exist"
Nov 02 15:55:31.573637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.573574    5041 scope.go:122] "RemoveContainer" containerID="a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad"
Nov 02 15:55:31.582622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.582600    5041 scope.go:122] "RemoveContainer" containerID="a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad"
Nov 02 15:55:31.582952 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:31.582918    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad\": container with ID starting with a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad not found: ID does not exist" containerID="a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad"
Nov 02 15:55:31.583043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.582961    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad"} err="failed to get container status \"a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad\": rpc error: code = NotFound desc = could not find container \"a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad\": container with ID starting with a0034c28621f5bbb0ab42fed04f83838f0cf8fc93624e81a4964184b5ce9f0ad not found: ID does not exist"
Nov 02 15:55:31.626894 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.626868    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"]
Nov 02 15:55:31.632834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.632811    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce791-645c8bcfc5-q47zj"]
Nov 02 15:55:31.671236 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.671210    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"]
Nov 02 15:55:31.677312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.677288    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce791-86bdc78d97-sp7zb"]
Nov 02 15:55:31.693722 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.693701    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:31.784952 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.784924    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4\" (UniqueName: \"kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4\") pod \"5f01110d-87a9-467f-95be-4c4195a78793\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") "
Nov 02 15:55:31.785074 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.785027    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca\") pod \"5f01110d-87a9-467f-95be-4c4195a78793\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") "
Nov 02 15:55:31.785141 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.785074    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca\") pod \"5f01110d-87a9-467f-95be-4c4195a78793\" (UID: \"5f01110d-87a9-467f-95be-4c4195a78793\") "
Nov 02 15:55:31.785382 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.785336    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca" pod "5f01110d-87a9-467f-95be-4c4195a78793" (UID: "5f01110d-87a9-467f-95be-4c4195a78793"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:31.786941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.786918    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4" pod "5f01110d-87a9-467f-95be-4c4195a78793" (UID: "5f01110d-87a9-467f-95be-4c4195a78793"). InnerVolumeSpecName "kube-api-access-ctxz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:31.787032 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.787004    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca" pod "5f01110d-87a9-467f-95be-4c4195a78793" (UID: "5f01110d-87a9-467f-95be-4c4195a78793"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:31.885786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.885728    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" path="/var/lib/kubelet/pods/50a1a0fb-bea2-4061-b32c-67ed9afeb526/volumes"
Nov 02 15:55:31.885931 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.885913    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxz4\" (UniqueName: \"kubernetes.io/projected/5f01110d-87a9-467f-95be-4c4195a78793-kube-api-access-ctxz4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.885980 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.885934    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.886017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.885999    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f01110d-87a9-467f-95be-4c4195a78793-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.886136 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:31.886123    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" path="/var/lib/kubelet/pods/fd6539ec-29c7-41d1-ba11-0d1a920530ae/volumes"
Nov 02 15:55:32.568103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:32.568079    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"
Nov 02 15:55:32.568565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:32.568126    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf" event={"ID":"5f01110d-87a9-467f-95be-4c4195a78793","Type":"ContainerDied","Data":"f16f4de71913c7b7a22968820ce9c7e84188f1ac03e05e1632aa02c609b3c519"}
Nov 02 15:55:32.568565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:32.568186    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f16f4de71913c7b7a22968820ce9c7e84188f1ac03e05e1632aa02c609b3c519"
Nov 02 15:55:32.869672 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:32.869590    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" podUID="87376fee-2fff-4e42-af60-c78478971f37" containerName="payload-processing" containerID="cri-o://dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113" gracePeriod=25
Nov 02 15:55:32.906511 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:32.906478    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerName="payload-pre-processing" containerID="cri-o://c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3" gracePeriod=25
Nov 02 15:55:34.402292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:34.402257    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" podUID="87376fee-2fff-4e42-af60-c78478971f37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:34.402705 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:34.402602    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:36.766980 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:36.766937    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:55:37.048144 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.048053    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main" containerID="cri-o://a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc" gracePeriod=45
Nov 02 15:55:37.295681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.295654    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:55:37.326463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326400    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs\") pod \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") "
Nov 02 15:55:37.326463 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326463    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home\") pod \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") "
Nov 02 15:55:37.326651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326489    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q\" (UniqueName: \"kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q\") pod \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") "
Nov 02 15:55:37.326651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326526    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir\") pod \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") "
Nov 02 15:55:37.326651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326554    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm\") pod \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") "
Nov 02 15:55:37.326651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326616    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache\") pod \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\" (UID: \"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b\") "
Nov 02 15:55:37.326895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326788    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir" pod "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" (UID: "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.326895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326807    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home" pod "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" (UID: "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.327010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326938    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache" pod "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" (UID: "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.327010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326961    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.327010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.326980    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.328725 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.328692    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm" pod "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" (UID: "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.328821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.328723    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs" pod "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" (UID: "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:37.328821 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.328768    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q" pod "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" (UID: "e3c359bc-c443-43b2-b9a7-2f12d2c42b8b"). InnerVolumeSpecName "kube-api-access-p747q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:37.428113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.428070    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.428113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.428112    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.428405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.428129    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p747q\" (UniqueName: \"kubernetes.io/projected/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-kube-api-access-p747q\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.428405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.428145    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.599858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.599766    5041 generic.go:356] "Generic (PLEG): container finished" podID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerID="a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc" exitCode=0
Nov 02 15:55:37.599858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.599813    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"
Nov 02 15:55:37.600077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.599852    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" event={"ID":"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b","Type":"ContainerDied","Data":"a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc"}
Nov 02 15:55:37.600077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.599897    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g" event={"ID":"e3c359bc-c443-43b2-b9a7-2f12d2c42b8b","Type":"ContainerDied","Data":"c78aa77ac37d1b8b3e1562daca2d7d869f7ce80e6d02218fa3b5967360fe4ae7"}
Nov 02 15:55:37.600077 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.599914    5041 scope.go:122] "RemoveContainer" containerID="a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc"
Nov 02 15:55:37.610453 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.610347    5041 scope.go:122] "RemoveContainer" containerID="a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc"
Nov 02 15:55:37.610663 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:37.610642    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc\": container with ID starting with a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc not found: ID does not exist" containerID="a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc"
Nov 02 15:55:37.610710 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.610671    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc"} err="failed to get container status \"a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc\": rpc error: code = NotFound desc = could not find container \"a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc\": container with ID starting with a368008abd088a91a1f69e068d415ada67906b3d6eac2371f1398db29a85a0fc not found: ID does not exist"
Nov 02 15:55:37.649865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.649838    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"]
Nov 02 15:55:37.661145 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.661125    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-2-w1-5dd8e9-kserve-6cb5b47944-8rx2g"]
Nov 02 15:55:37.885842 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:37.885780    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" path="/var/lib/kubelet/pods/e3c359bc-c443-43b2-b9a7-2f12d2c42b8b/volumes"
Nov 02 15:55:43.901465 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:43.901432    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"]
Nov 02 15:55:43.912687 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:43.912653    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5dd8e9-xrx5t"]
Nov 02 15:55:44.402780 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:44.402742    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:44.402985 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:44.402740    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" podUID="87376fee-2fff-4e42-af60-c78478971f37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:55:45.886149 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:45.886113    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b977e400-c791-4b0f-a753-7ae2094d2742" path="/var/lib/kubelet/pods/b977e400-c791-4b0f-a753-7ae2094d2742/volumes"
Nov 02 15:55:49.359301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:49.359203    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:55:49.383154 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:49.369955    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 15:55:53.188169 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.188144    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:55:53.201589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.201568    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:55:53.263258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263230    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume\") pod \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") "
Nov 02 15:55:53.263401 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263313    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd\" (UniqueName: \"kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd\") pod \"87376fee-2fff-4e42-af60-c78478971f37\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") "
Nov 02 15:55:53.263480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263420    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume\") pod \"87376fee-2fff-4e42-af60-c78478971f37\" (UID: \"87376fee-2fff-4e42-af60-c78478971f37\") "
Nov 02 15:55:53.263554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263498    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m\" (UniqueName: \"kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m\") pod \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\" (UID: \"9c2b628b-5a9a-4990-bd39-26bd0a01cca0\") "
Nov 02 15:55:53.263796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263769    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume" pod "87376fee-2fff-4e42-af60-c78478971f37" (UID: "87376fee-2fff-4e42-af60-c78478971f37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:53.263906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263769    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume" pod "9c2b628b-5a9a-4990-bd39-26bd0a01cca0" (UID: "9c2b628b-5a9a-4990-bd39-26bd0a01cca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:53.263969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263916    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:53.263969 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.263938    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87376fee-2fff-4e42-af60-c78478971f37-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:53.265330 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.265305    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd" pod "87376fee-2fff-4e42-af60-c78478971f37" (UID: "87376fee-2fff-4e42-af60-c78478971f37"). InnerVolumeSpecName "kube-api-access-b66rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:53.265630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.265612    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m" pod "9c2b628b-5a9a-4990-bd39-26bd0a01cca0" (UID: "9c2b628b-5a9a-4990-bd39-26bd0a01cca0"). InnerVolumeSpecName "kube-api-access-7hl4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:53.364928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.364903    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b66rd\" (UniqueName: \"kubernetes.io/projected/87376fee-2fff-4e42-af60-c78478971f37-kube-api-access-b66rd\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:53.364928 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.364927    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hl4m\" (UniqueName: \"kubernetes.io/projected/9c2b628b-5a9a-4990-bd39-26bd0a01cca0-kube-api-access-7hl4m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:55:53.689958 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.689927    5041 generic.go:356] "Generic (PLEG): container finished" podID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerID="c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3" exitCode=0
Nov 02 15:55:53.690105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.689969    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"
Nov 02 15:55:53.690105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.690012    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" event={"ID":"9c2b628b-5a9a-4990-bd39-26bd0a01cca0","Type":"ContainerDied","Data":"c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3"}
Nov 02 15:55:53.690105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.690065    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt" event={"ID":"9c2b628b-5a9a-4990-bd39-26bd0a01cca0","Type":"ContainerDied","Data":"d1679b26025b6dde3532702fdd89b57e206ef2068ee0b29af5af35bfcbd79344"}
Nov 02 15:55:53.690105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.690091    5041 scope.go:122] "RemoveContainer" containerID="c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3"
Nov 02 15:55:53.691455 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.691436    5041 generic.go:356] "Generic (PLEG): container finished" podID="87376fee-2fff-4e42-af60-c78478971f37" containerID="dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113" exitCode=0
Nov 02 15:55:53.691615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.691475    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" event={"ID":"87376fee-2fff-4e42-af60-c78478971f37","Type":"ContainerDied","Data":"dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113"}
Nov 02 15:55:53.691615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.691487    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"
Nov 02 15:55:53.691615 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.691503    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd" event={"ID":"87376fee-2fff-4e42-af60-c78478971f37","Type":"ContainerDied","Data":"aecd7b2f6f109a5e1f3922e3ad375d77df7e90578b07129f17697f1ba63e466d"}
Nov 02 15:55:53.701213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.701198    5041 scope.go:122] "RemoveContainer" containerID="c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3"
Nov 02 15:55:53.701606 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:53.701578    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3\": container with ID starting with c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3 not found: ID does not exist" containerID="c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3"
Nov 02 15:55:53.701681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.701620    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3"} err="failed to get container status \"c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3\": rpc error: code = NotFound desc = could not find container \"c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3\": container with ID starting with c0f8e78b14b1ba9b53212dcc00600248b3e62beb2e98738f91702c3fe82276f3 not found: ID does not exist"
Nov 02 15:55:53.701681 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.701644    5041 scope.go:122] "RemoveContainer" containerID="dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113"
Nov 02 15:55:53.710950 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.710934    5041 scope.go:122] "RemoveContainer" containerID="dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113"
Nov 02 15:55:53.711201 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:55:53.711181    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113\": container with ID starting with dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113 not found: ID does not exist" containerID="dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113"
Nov 02 15:55:53.711266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.711208    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113"} err="failed to get container status \"dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113\": rpc error: code = NotFound desc = could not find container \"dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113\": container with ID starting with dedc7695f948dbc847aa7b5cd30ed3349a72746d3dce4fab07ebb1f4d406e113 not found: ID does not exist"
Nov 02 15:55:53.785724 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.785699    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"]
Nov 02 15:55:53.801163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.801141    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5dd8e9-5c5fbff9d7-zhjwd"]
Nov 02 15:55:53.837734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.837714    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"]
Nov 02 15:55:53.841922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.841900    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5dd8e9-5c7d8fb6c9-8smqt"]
Nov 02 15:55:53.887844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.887824    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87376fee-2fff-4e42-af60-c78478971f37" path="/var/lib/kubelet/pods/87376fee-2fff-4e42-af60-c78478971f37/volumes"
Nov 02 15:55:53.888205 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:55:53.888191    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" path="/var/lib/kubelet/pods/9c2b628b-5a9a-4990-bd39-26bd0a01cca0/volumes"
Nov 02 15:56:06.768723 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:56:06.768677    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:56:27.132052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.132016    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hvszm"]
Nov 02 15:56:27.132614 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.132244    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6667f79cdf-hvszm" podUID="ce27d7ab-8df4-47df-8564-a18e220911f6" containerName="manager" containerID="cri-o://d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e" gracePeriod=30
Nov 02 15:56:27.386901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.386851    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:56:27.460290 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.460255    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl\" (UniqueName: \"kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl\") pod \"ce27d7ab-8df4-47df-8564-a18e220911f6\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") "
Nov 02 15:56:27.460462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.460394    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs\") pod \"ce27d7ab-8df4-47df-8564-a18e220911f6\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") "
Nov 02 15:56:27.460535 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.460499    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert\") pod \"ce27d7ab-8df4-47df-8564-a18e220911f6\" (UID: \"ce27d7ab-8df4-47df-8564-a18e220911f6\") "
Nov 02 15:56:27.462557 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.462522    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs" pod "ce27d7ab-8df4-47df-8564-a18e220911f6" (UID: "ce27d7ab-8df4-47df-8564-a18e220911f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:56:27.462848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.462822    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert" pod "ce27d7ab-8df4-47df-8564-a18e220911f6" (UID: "ce27d7ab-8df4-47df-8564-a18e220911f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:56:27.462942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.462907    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl" pod "ce27d7ab-8df4-47df-8564-a18e220911f6" (UID: "ce27d7ab-8df4-47df-8564-a18e220911f6"). InnerVolumeSpecName "kube-api-access-hbjcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:56:27.561266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.561239    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:56:27.561266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.561262    5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce27d7ab-8df4-47df-8564-a18e220911f6-webhook-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:56:27.561473 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.561274    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjcl\" (UniqueName: \"kubernetes.io/projected/ce27d7ab-8df4-47df-8564-a18e220911f6-kube-api-access-hbjcl\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:56:27.882503 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.882472    5041 generic.go:356] "Generic (PLEG): container finished" podID="ce27d7ab-8df4-47df-8564-a18e220911f6" containerID="d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e" exitCode=0
Nov 02 15:56:27.883960 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.883916    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-hvszm"
Nov 02 15:56:27.888279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.888247    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-hvszm" event={"ID":"ce27d7ab-8df4-47df-8564-a18e220911f6","Type":"ContainerDied","Data":"d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e"}
Nov 02 15:56:27.888419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.888304    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-hvszm" event={"ID":"ce27d7ab-8df4-47df-8564-a18e220911f6","Type":"ContainerDied","Data":"4f1a9dc29f4ab47a5545dabbaa22d9d0631a2c8a5c960a9320a6209ed62e5b1a"}
Nov 02 15:56:27.888419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.888337    5041 scope.go:122] "RemoveContainer" containerID="d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e"
Nov 02 15:56:27.900238 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.900221    5041 scope.go:122] "RemoveContainer" containerID="d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e"
Nov 02 15:56:27.900539 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:56:27.900517    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e\": container with ID starting with d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e not found: ID does not exist" containerID="d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e"
Nov 02 15:56:27.900618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.900552    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e"} err="failed to get container status \"d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e\": rpc error: code = NotFound desc = could not find container \"d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e\": container with ID starting with d94bdb3dc3e0303f541121f61ebd17060ea28c86677798e72fb915c8aebff43e not found: ID does not exist"
Nov 02 15:56:27.933848 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.933820    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hvszm"]
Nov 02 15:56:27.945667 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:27.945639    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hvszm"]
Nov 02 15:56:28.617626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.617599    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6667f79cdf-86rkg"]
Nov 02 15:56:28.618013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.617974    5041 cpu_manager.go:414] "containerMap: removing container" podUID="87376fee-2fff-4e42-af60-c78478971f37" containerName="payload-processing"
Nov 02 15:56:28.618013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.617987    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87376fee-2fff-4e42-af60-c78478971f37" containerName="payload-processing"
Nov 02 15:56:28.618013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.617999    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerName="payload-pre-processing"
Nov 02 15:56:28.618013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618004    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerName="payload-pre-processing"
Nov 02 15:56:28.618013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618014    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ce27d7ab-8df4-47df-8564-a18e220911f6" containerName="manager"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618019    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce27d7ab-8df4-47df-8564-a18e220911f6" containerName="manager"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618026    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618031    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618038    5041 cpu_manager.go:414] "containerMap: removing container" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerName="payload-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618042    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerName="payload-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618049    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f01110d-87a9-467f-95be-4c4195a78793" containerName="revoke-keys"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618054    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f01110d-87a9-467f-95be-4c4195a78793" containerName="revoke-keys"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618077    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerName="payload-pre-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618082    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerName="payload-pre-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618147    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2b628b-5a9a-4990-bd39-26bd0a01cca0" containerName="payload-pre-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618157    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="87376fee-2fff-4e42-af60-c78478971f37" containerName="payload-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618164    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6539ec-29c7-41d1-ba11-0d1a920530ae" containerName="payload-pre-processing"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618171    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce27d7ab-8df4-47df-8564-a18e220911f6" containerName="manager"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618179    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c359bc-c443-43b2-b9a7-2f12d2c42b8b" containerName="main"
Nov 02 15:56:28.618189 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618187    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a1a0fb-bea2-4061-b32c-67ed9afeb526" containerName="payload-processing"
Nov 02 15:56:28.618867 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.618197    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f01110d-87a9-467f-95be-4c4195a78793" containerName="revoke-keys"
Nov 02 15:56:28.622841 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.622823    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.649525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.649472    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 15:56:28.649525 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.649497    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 15:56:28.649691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.649497    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kr787\""
Nov 02 15:56:28.649691 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.649555    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 15:56:28.651942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.651921    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-86rkg"]
Nov 02 15:56:28.669468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.669445    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnjq\" (UniqueName: \"kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.669555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.669504    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.669555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.669526    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.770312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.770285    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnjq\" (UniqueName: \"kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.770462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.770340    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.770462 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.770389    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.772908 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.772882    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.772988 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.772910    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.792628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.792603    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnjq\" (UniqueName: \"kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq\") pod \"maas-controller-6667f79cdf-86rkg\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") " pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.893607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.893544    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"]
Nov 02 15:56:28.897337 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.897318    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.923610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.923586    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"]
Nov 02 15:56:28.936258 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.936236    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:28.972196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972169    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972222    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972421 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972336    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972424    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972564 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972549    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972588    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972645 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972622    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972823 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972790    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcxd\" (UniqueName: \"kubernetes.io/projected/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-kube-api-access-4bcxd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:28.972870 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:28.972849    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073513 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073480    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073553    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073590    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073627    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073669    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcxd\" (UniqueName: \"kubernetes.io/projected/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-kube-api-access-4bcxd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073695    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073727    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.073813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073791    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.074049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073841    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.074049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.073976    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.074226 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.074200    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.074352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.074304    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.074352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.074332    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.074579 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.074556    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.075906 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.075877    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.076062 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.076045    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.096103 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.096080    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.096747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.096730    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcxd\" (UniqueName: \"kubernetes.io/projected/03da44d6-bcfd-4e9f-88d7-a9b27157d56b-kube-api-access-4bcxd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rdl5t\" (UID: \"03da44d6-bcfd-4e9f-88d7-a9b27157d56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.211400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.211373    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:29.238520 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.238490    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-86rkg"]
Nov 02 15:56:29.240439 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:56:29.240407    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644236e1_6599_4221_b03c_810068154cfc.slice/crio-310ab2b4ebe387c6c455e2182d85efed8c468c3a998f0351cd174640d40d0295 WatchSource:0}: Error finding container 310ab2b4ebe387c6c455e2182d85efed8c468c3a998f0351cd174640d40d0295: Status 404 returned error can't find the container with id 310ab2b4ebe387c6c455e2182d85efed8c468c3a998f0351cd174640d40d0295
Nov 02 15:56:29.241674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.241655    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:56:29.523063 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.523033    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"]
Nov 02 15:56:29.524199 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:56:29.524170    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03da44d6_bcfd_4e9f_88d7_a9b27157d56b.slice/crio-9cdc6ef9cb6755209f6ab371ec9e5b86a73b3a4fe71fe6c2a55de848ebd644d4 WatchSource:0}: Error finding container 9cdc6ef9cb6755209f6ab371ec9e5b86a73b3a4fe71fe6c2a55de848ebd644d4: Status 404 returned error can't find the container with id 9cdc6ef9cb6755209f6ab371ec9e5b86a73b3a4fe71fe6c2a55de848ebd644d4
Nov 02 15:56:29.526530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.526487    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:56:29.526627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.526555    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:56:29.526627 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.526588    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:56:29.887476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.887442    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce27d7ab-8df4-47df-8564-a18e220911f6" path="/var/lib/kubelet/pods/ce27d7ab-8df4-47df-8564-a18e220911f6/volumes"
Nov 02 15:56:29.895413 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.895358    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-86rkg" event={"ID":"644236e1-6599-4221-b03c-810068154cfc","Type":"ContainerStarted","Data":"6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"}
Nov 02 15:56:29.895542 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.895439    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-86rkg" event={"ID":"644236e1-6599-4221-b03c-810068154cfc","Type":"ContainerStarted","Data":"310ab2b4ebe387c6c455e2182d85efed8c468c3a998f0351cd174640d40d0295"}
Nov 02 15:56:29.895603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.895558    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:29.897109 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.897085    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" event={"ID":"03da44d6-bcfd-4e9f-88d7-a9b27157d56b","Type":"ContainerStarted","Data":"3dd4769c5a7360c55e272da3db5e6c1a7cb221cabf808084455d1aba5b94138b"}
Nov 02 15:56:29.897206 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.897133    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" event={"ID":"03da44d6-bcfd-4e9f-88d7-a9b27157d56b","Type":"ContainerStarted","Data":"9cdc6ef9cb6755209f6ab371ec9e5b86a73b3a4fe71fe6c2a55de848ebd644d4"}
Nov 02 15:56:29.948389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.943033    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6667f79cdf-86rkg" podStartSLOduration=1.463589105 podStartE2EDuration="1.943013866s" totalImagesPullingTime="479.424761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:29.941208365 +0000 UTC m=+3642.199277815" watchObservedRunningTime="2026-11-02 15:56:29.943013866 +0000 UTC m=+3642.201083318"
Nov 02 15:56:29.988163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:29.988111    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podStartSLOduration=1.98809732 podStartE2EDuration="1.98809732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:29.986181726 +0000 UTC m=+3642.244251175" watchObservedRunningTime="2026-11-02 15:56:29.98809732 +0000 UTC m=+3642.246166770"
Nov 02 15:56:30.212186 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:30.212151    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:30.217391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:30.213926    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:30.217391 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:30.213989    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:31.212327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:31.212291    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:31.212897 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:31.212351    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:32.216390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:32.212749    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:32.216390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:32.212810    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:33.212057 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:33.212022    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:33.212327 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:33.212076    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:34.211767 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:34.211732    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:34.212266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:34.211794    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:35.212104 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:35.212068    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:35.212611 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:35.212124    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:36.212470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:36.212431    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rdl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body=
Nov 02 15:56:36.212888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:36.212484    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t" podUID="03da44d6-bcfd-4e9f-88d7-a9b27157d56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused"
Nov 02 15:56:36.769880 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:56:36.769828    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:56:37.657745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:37.657709    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:37.658215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:37.658058    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:37.658972 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:37.658949    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rdl5t"
Nov 02 15:56:39.972862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:39.972836    5041 generic.go:356] "Generic (PLEG): container finished" podID="644236e1-6599-4221-b03c-810068154cfc" containerID="6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650" exitCode=1
Nov 02 15:56:39.973196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:39.972914    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-86rkg" event={"ID":"644236e1-6599-4221-b03c-810068154cfc","Type":"ContainerDied","Data":"6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"}
Nov 02 15:56:39.973269 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:39.973250    5041 scope.go:122] "RemoveContainer" containerID="6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"
Nov 02 15:56:40.980129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:40.980093    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-86rkg" event={"ID":"644236e1-6599-4221-b03c-810068154cfc","Type":"ContainerStarted","Data":"f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9"}
Nov 02 15:56:40.980565 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:40.980287    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:51.988199 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:51.988171    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:56:59.680697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.680651    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"]
Nov 02 15:56:59.685660 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.685631    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.714458 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.704680    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:56:59.719670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.719484    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:56:59.719670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.719571    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:56:59.719670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.719645    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:56:59.729203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.729175    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"]
Nov 02 15:56:59.764343 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.764318    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.764531 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.764513    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmjz\" (UniqueName: \"kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.764602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.764566    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.865529 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.865496    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmjz\" (UniqueName: \"kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.865679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.865539    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.865679 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.865581    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.866240 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.866208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.867898 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.867877    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:56:59.888426 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:56:59.888402    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmjz\" (UniqueName: \"kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz\") pod \"maas-api-revoke-sub-85237e45-h42mx\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:57:00.020064 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:00.019979    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:57:00.322047 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:00.322015    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"]
Nov 02 15:57:00.323616 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:57:00.323591    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6311d1_f26b_41b2_b68a_23a8f309d712.slice/crio-453719cf557362a457c62d95a594cc6eb549c9b6054f655c6e0c88ee3b201cc5 WatchSource:0}: Error finding container 453719cf557362a457c62d95a594cc6eb549c9b6054f655c6e0c88ee3b201cc5: Status 404 returned error can't find the container with id 453719cf557362a457c62d95a594cc6eb549c9b6054f655c6e0c88ee3b201cc5
Nov 02 15:57:01.101444 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:01.101414    5041 generic.go:356] "Generic (PLEG): container finished" podID="6e6311d1-f26b-41b2-b68a-23a8f309d712" containerID="e0c0e9abc986517d749e17127101907905e4e028a10668e81dab613a77abaaec" exitCode=0
Nov 02 15:57:01.101803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:01.101512    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx" event={"ID":"6e6311d1-f26b-41b2-b68a-23a8f309d712","Type":"ContainerDied","Data":"e0c0e9abc986517d749e17127101907905e4e028a10668e81dab613a77abaaec"}
Nov 02 15:57:01.101803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:01.101566    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx" event={"ID":"6e6311d1-f26b-41b2-b68a-23a8f309d712","Type":"ContainerStarted","Data":"453719cf557362a457c62d95a594cc6eb549c9b6054f655c6e0c88ee3b201cc5"}
Nov 02 15:57:02.281658 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.281628    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:57:02.389148 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.389057    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz\" (UniqueName: \"kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz\") pod \"6e6311d1-f26b-41b2-b68a-23a8f309d712\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") "
Nov 02 15:57:02.389328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.389178    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca\") pod \"6e6311d1-f26b-41b2-b68a-23a8f309d712\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") "
Nov 02 15:57:02.389328 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.389261    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca\") pod \"6e6311d1-f26b-41b2-b68a-23a8f309d712\" (UID: \"6e6311d1-f26b-41b2-b68a-23a8f309d712\") "
Nov 02 15:57:02.389657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.389620    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca" pod "6e6311d1-f26b-41b2-b68a-23a8f309d712" (UID: "6e6311d1-f26b-41b2-b68a-23a8f309d712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:57:02.391193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.391167    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz" pod "6e6311d1-f26b-41b2-b68a-23a8f309d712" (UID: "6e6311d1-f26b-41b2-b68a-23a8f309d712"). InnerVolumeSpecName "kube-api-access-qvmjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:57:02.391312 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.391229    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca" pod "6e6311d1-f26b-41b2-b68a-23a8f309d712" (UID: "6e6311d1-f26b-41b2-b68a-23a8f309d712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:57:02.490345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.490307    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:57:02.490345 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.490342    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6311d1-f26b-41b2-b68a-23a8f309d712-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:57:02.490572 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:02.490356    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmjz\" (UniqueName: \"kubernetes.io/projected/6e6311d1-f26b-41b2-b68a-23a8f309d712-kube-api-access-qvmjz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:57:03.121737 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:03.121695    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx" event={"ID":"6e6311d1-f26b-41b2-b68a-23a8f309d712","Type":"ContainerDied","Data":"453719cf557362a457c62d95a594cc6eb549c9b6054f655c6e0c88ee3b201cc5"}
Nov 02 15:57:03.121986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:03.121763    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453719cf557362a457c62d95a594cc6eb549c9b6054f655c6e0c88ee3b201cc5"
Nov 02 15:57:03.121986 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:03.121837    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"
Nov 02 15:57:06.767115 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:57:06.767081    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:57:11.497521 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.497487    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-86rkg"]
Nov 02 15:57:11.498017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.497708    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6667f79cdf-86rkg" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager" containerID="cri-o://f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9" gracePeriod=30
Nov 02 15:57:11.749568 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.749498    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:57:11.869241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.869210    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs\") pod \"644236e1-6599-4221-b03c-810068154cfc\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") "
Nov 02 15:57:11.869512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.869420    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert\") pod \"644236e1-6599-4221-b03c-810068154cfc\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") "
Nov 02 15:57:11.869512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.869460    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq\" (UniqueName: \"kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq\") pod \"644236e1-6599-4221-b03c-810068154cfc\" (UID: \"644236e1-6599-4221-b03c-810068154cfc\") "
Nov 02 15:57:11.871508 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.871473    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert" pod "644236e1-6599-4221-b03c-810068154cfc" (UID: "644236e1-6599-4221-b03c-810068154cfc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:57:11.871628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.871547    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq" pod "644236e1-6599-4221-b03c-810068154cfc" (UID: "644236e1-6599-4221-b03c-810068154cfc"). InnerVolumeSpecName "kube-api-access-tdnjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:57:11.871628 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.871543    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs" pod "644236e1-6599-4221-b03c-810068154cfc" (UID: "644236e1-6599-4221-b03c-810068154cfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:57:11.970774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.970749    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:57:11.970774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.970773    5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644236e1-6599-4221-b03c-810068154cfc-webhook-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:57:11.970932 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:11.970782    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnjq\" (UniqueName: \"kubernetes.io/projected/644236e1-6599-4221-b03c-810068154cfc-kube-api-access-tdnjq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:57:12.176138 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.176104    5041 generic.go:356] "Generic (PLEG): container finished" podID="644236e1-6599-4221-b03c-810068154cfc" containerID="f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9" exitCode=0
Nov 02 15:57:12.176315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.176153    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-86rkg"
Nov 02 15:57:12.176315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.176165    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-86rkg" event={"ID":"644236e1-6599-4221-b03c-810068154cfc","Type":"ContainerDied","Data":"f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9"}
Nov 02 15:57:12.176315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.176222    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-86rkg" event={"ID":"644236e1-6599-4221-b03c-810068154cfc","Type":"ContainerDied","Data":"310ab2b4ebe387c6c455e2182d85efed8c468c3a998f0351cd174640d40d0295"}
Nov 02 15:57:12.176315 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.176249    5041 scope.go:122] "RemoveContainer" containerID="f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9"
Nov 02 15:57:12.185609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.185589    5041 scope.go:122] "RemoveContainer" containerID="6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"
Nov 02 15:57:12.193941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.193923    5041 scope.go:122] "RemoveContainer" containerID="f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9"
Nov 02 15:57:12.194179 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:57:12.194160    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9\": container with ID starting with f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9 not found: ID does not exist" containerID="f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9"
Nov 02 15:57:12.194230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.194186    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9"} err="failed to get container status \"f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9\": rpc error: code = NotFound desc = could not find container \"f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9\": container with ID starting with f4e6d2b05f4936386a2d9bd5f03038a7318d466c1b6b8ea8b47b295fe37766d9 not found: ID does not exist"
Nov 02 15:57:12.194230 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.194205    5041 scope.go:122] "RemoveContainer" containerID="6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"
Nov 02 15:57:12.194474 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:57:12.194454    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650\": container with ID starting with 6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650 not found: ID does not exist" containerID="6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"
Nov 02 15:57:12.194543 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.194483    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650"} err="failed to get container status \"6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650\": rpc error: code = NotFound desc = could not find container \"6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650\": container with ID starting with 6f4cb64fb5ee533a50a1d2650798f7d523b3bc6079903e155376d4dde531f650 not found: ID does not exist"
Nov 02 15:57:12.227501 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.227479    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-86rkg"]
Nov 02 15:57:12.231622 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:12.231601    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-86rkg"]
Nov 02 15:57:13.572044 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572001    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hj9ms"]
Nov 02 15:57:13.572644 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572621    5041 cpu_manager.go:414] "containerMap: removing container" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager"
Nov 02 15:57:13.572745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572645    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager"
Nov 02 15:57:13.572745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572708    5041 cpu_manager.go:414] "containerMap: removing container" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager"
Nov 02 15:57:13.572745 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572718    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager"
Nov 02 15:57:13.572910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572746    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6e6311d1-f26b-41b2-b68a-23a8f309d712" containerName="revoke-keys"
Nov 02 15:57:13.572910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572755    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6311d1-f26b-41b2-b68a-23a8f309d712" containerName="revoke-keys"
Nov 02 15:57:13.572910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572885    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager"
Nov 02 15:57:13.572910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.572901    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6311d1-f26b-41b2-b68a-23a8f309d712" containerName="revoke-keys"
Nov 02 15:57:13.577405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.577359    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.591796 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.591773    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 15:57:13.603083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.603053    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kr787\""
Nov 02 15:57:13.603277 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.603256    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 15:57:13.603350 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.603257    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 15:57:13.606428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.606403    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hj9ms"]
Nov 02 15:57:13.686570 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.686534    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-metrics-certs\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.686715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.686662    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv2g\" (UniqueName: \"kubernetes.io/projected/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-kube-api-access-tjv2g\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.686781 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.686716    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-webhook-cert\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.787224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.787180    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-webhook-cert\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.787456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.787255    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-metrics-certs\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.787456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.787313    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv2g\" (UniqueName: \"kubernetes.io/projected/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-kube-api-access-tjv2g\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.789942 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.789905    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-webhook-cert\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.790053 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.789902    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-metrics-certs\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.810922 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.810893    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv2g\" (UniqueName: \"kubernetes.io/projected/364c4ded-4de4-4cb2-8c46-d9f2e895dcf2-kube-api-access-tjv2g\") pod \"maas-controller-6667f79cdf-hj9ms\" (UID: \"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2\") " pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:13.886355 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.886278    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644236e1-6599-4221-b03c-810068154cfc" path="/var/lib/kubelet/pods/644236e1-6599-4221-b03c-810068154cfc/volumes"
Nov 02 15:57:13.890348 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:13.890325    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:14.189287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:14.189250    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6667f79cdf-hj9ms"]
Nov 02 15:57:14.189512 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:57:14.189486    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364c4ded_4de4_4cb2_8c46_d9f2e895dcf2.slice/crio-bf052e89c12d21b2bf31caae357778fac6cf25d339fbca38520f1e00dad93d5c WatchSource:0}: Error finding container bf052e89c12d21b2bf31caae357778fac6cf25d339fbca38520f1e00dad93d5c: Status 404 returned error can't find the container with id bf052e89c12d21b2bf31caae357778fac6cf25d339fbca38520f1e00dad93d5c
Nov 02 15:57:15.199026 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:15.198984    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-hj9ms" event={"ID":"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2","Type":"ContainerStarted","Data":"bfa59f8c05983aa3c415ffc97feef91e2976e6258a73ed82caab636bcaac6cca"}
Nov 02 15:57:15.199470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:15.199051    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6667f79cdf-hj9ms" event={"ID":"364c4ded-4de4-4cb2-8c46-d9f2e895dcf2","Type":"ContainerStarted","Data":"bf052e89c12d21b2bf31caae357778fac6cf25d339fbca38520f1e00dad93d5c"}
Nov 02 15:57:15.199470 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:15.199074    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:15.241913 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:15.241859    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6667f79cdf-hj9ms" podStartSLOduration=1.945047523 podStartE2EDuration="2.241845441s" totalImagesPullingTime="296.797918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:15.241272061 +0000 UTC m=+3687.499341535" watchObservedRunningTime="2026-11-02 15:57:15.241845441 +0000 UTC m=+3687.499914916"
Nov 02 15:57:26.207494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:26.207460    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6667f79cdf-hj9ms"
Nov 02 15:57:29.069707 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.069671    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"]
Nov 02 15:57:29.070182 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.070168    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="644236e1-6599-4221-b03c-810068154cfc" containerName="manager"
Nov 02 15:57:29.074309 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.074286    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.103803 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.103776    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"]
Nov 02 15:57:29.107606 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.107586    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.109476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109449    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7z9\" (UniqueName: \"kubernetes.io/projected/23e8c095-a07b-4484-9fdf-0c87e5392cb8-kube-api-access-2z7z9\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109494    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109524    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109565    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109747 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109623    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109850 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109825    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109875    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109946 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109913    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.109981 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.109947    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.122163 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.122134    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"]
Nov 02 15:57:29.143497 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.143474    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"]
Nov 02 15:57:29.210815 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.210785    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.210953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.210832    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.210953 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.210941    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.210973    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211083 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.210992    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211113    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211149    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211170    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211183    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211206    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211229    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7z9\" (UniqueName: \"kubernetes.io/projected/23e8c095-a07b-4484-9fdf-0c87e5392cb8-kube-api-access-2z7z9\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211279    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211299    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211326    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211351    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211394    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4w2\" (UniqueName: \"kubernetes.io/projected/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-kube-api-access-vl4w2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211468 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211426    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211473    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211510    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.211904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211635    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211731    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211802    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.211904 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.211820    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.213970 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.213950    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.214135 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.214117    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.234336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.234312    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e8c095-a07b-4484-9fdf-0c87e5392cb8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.234527 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.234506    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7z9\" (UniqueName: \"kubernetes.io/projected/23e8c095-a07b-4484-9fdf-0c87e5392cb8-kube-api-access-2z7z9\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k4x2d\" (UID: \"23e8c095-a07b-4484-9fdf-0c87e5392cb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.312927 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.312905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.312942    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.312966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313018 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.312982    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313019    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313038    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313179    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313218    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4w2\" (UniqueName: \"kubernetes.io/projected/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-kube-api-access-vl4w2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313405 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313250    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313437    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313669 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313645    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313730    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313813 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313770    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.313862 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.313803    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.315156 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.315133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.315358 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.315342    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.336020 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.335951    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.345696 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.345672    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4w2\" (UniqueName: \"kubernetes.io/projected/97ddb9f7-6726-49f3-9acc-c5602bcf68d1-kube-api-access-vl4w2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-khzpx\" (UID: \"97ddb9f7-6726-49f3-9acc-c5602bcf68d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.387439 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.387417    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:29.422630 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.422603    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:29.701754 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.701725    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"]
Nov 02 15:57:29.702945 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:57:29.702921    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e8c095_a07b_4484_9fdf_0c87e5392cb8.slice/crio-09b6238b29dbd24bf0fb3072b629ddfc99d4b464ad89d4dcde1c34c766025f17 WatchSource:0}: Error finding container 09b6238b29dbd24bf0fb3072b629ddfc99d4b464ad89d4dcde1c34c766025f17: Status 404 returned error can't find the container with id 09b6238b29dbd24bf0fb3072b629ddfc99d4b464ad89d4dcde1c34c766025f17
Nov 02 15:57:29.705101 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.705070    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.705160 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.705152    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.705202 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.705179    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.746190 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.746163    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"]
Nov 02 15:57:29.750014 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:57:29.749988    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ddb9f7_6726_49f3_9acc_c5602bcf68d1.slice/crio-6bdb46bacd34ddc4b4f2845b34ee053f5a912cce6092e8c104dddc7b16aad623 WatchSource:0}: Error finding container 6bdb46bacd34ddc4b4f2845b34ee053f5a912cce6092e8c104dddc7b16aad623: Status 404 returned error can't find the container with id 6bdb46bacd34ddc4b4f2845b34ee053f5a912cce6092e8c104dddc7b16aad623
Nov 02 15:57:29.752013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.751984    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.752089 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.752053    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.752129 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:29.752098    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:30.285825 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.285778    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" event={"ID":"97ddb9f7-6726-49f3-9acc-c5602bcf68d1","Type":"ContainerStarted","Data":"d6ee1662fe649456fafbb4d34bdc25ab6aea3aaac474a9e71181e2db9235dfe4"}
Nov 02 15:57:30.286287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.285849    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" event={"ID":"97ddb9f7-6726-49f3-9acc-c5602bcf68d1","Type":"ContainerStarted","Data":"6bdb46bacd34ddc4b4f2845b34ee053f5a912cce6092e8c104dddc7b16aad623"}
Nov 02 15:57:30.287336 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.287306    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" event={"ID":"23e8c095-a07b-4484-9fdf-0c87e5392cb8","Type":"ContainerStarted","Data":"6c863786467ca49e81b6a8f7a4a96d300e6d829ee68c94a35efbfaa9f81ef465"}
Nov 02 15:57:30.287471 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.287349    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" event={"ID":"23e8c095-a07b-4484-9fdf-0c87e5392cb8","Type":"ContainerStarted","Data":"09b6238b29dbd24bf0fb3072b629ddfc99d4b464ad89d4dcde1c34c766025f17"}
Nov 02 15:57:30.346390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.345855    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podStartSLOduration=1.345835046 podStartE2EDuration="1.345835046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:30.345405684 +0000 UTC m=+3702.603475134" watchObservedRunningTime="2026-11-02 15:57:30.345835046 +0000 UTC m=+3702.603904497"
Nov 02 15:57:30.387881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.387839    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:30.389272 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.389235    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:30.389428 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.389293    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:30.398624 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.398579    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podStartSLOduration=1.398567463 podStartE2EDuration="1.398567463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:30.397488516 +0000 UTC m=+3702.655557966" watchObservedRunningTime="2026-11-02 15:57:30.398567463 +0000 UTC m=+3702.656636907"
Nov 02 15:57:30.436386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.435598    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:30.436386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.435798    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:30.436386 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:30.435844    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:31.388674 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:31.388638    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:31.389078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:31.388687    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:31.426397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:31.425664    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:31.426397 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:31.425718    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:32.387766 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:32.387729    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:32.387935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:32.387782    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:32.436395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:32.435678    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:32.436395 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:32.435746    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:33.387865 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:33.387823    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:33.388086 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:33.387880    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:33.433881 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:33.433846    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:33.434200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:33.434174    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:34.388995 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:34.388953    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:34.389618 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:34.389579    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:34.436389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:34.433635    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:34.436389 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:34.433696    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:35.387858 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:35.387809    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:35.388052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:35.387877    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:35.426901 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:35.426864    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:35.427528 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:35.427496    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:36.388341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:36.388300    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k4x2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 15:57:36.388533 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:36.388388    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d" podUID="23e8c095-a07b-4484-9fdf-0c87e5392cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 15:57:36.423751 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:36.423710    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-khzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Nov 02 15:57:36.423888 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:36.423770    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx" podUID="97ddb9f7-6726-49f3-9acc-c5602bcf68d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Nov 02 15:57:36.769243 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:57:36.769199    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:57:38.003732 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:38.003679    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:38.004582 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:38.004539    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:38.006116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:38.006091    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k4x2d"
Nov 02 15:57:38.006292 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:38.006276    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:38.007173 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:38.007130    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:38.007810 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:38.007788    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-khzpx"
Nov 02 15:57:59.038649 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.038568    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"]
Nov 02 15:57:59.043593 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.043570    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.059384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.059355    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:57:59.059506 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.059492    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:57:59.059589 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.059524    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:57:59.069966 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.069945    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:57:59.072555 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.072527    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"]
Nov 02 15:57:59.175943 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.175916    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.176076 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.175954    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzqg\" (UniqueName: \"kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.176126 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.176093    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.276677 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.276651    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.276791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.276699    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.276791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.276730    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzqg\" (UniqueName: \"kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.277424 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.277406    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.278948 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.278929    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.299232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.299179    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzqg\" (UniqueName: \"kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg\") pod \"maas-api-revoke-sub-c1e271a9-2xk9t\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.356446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.356427    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:57:59.654245 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:57:59.654214    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"]
Nov 02 15:57:59.655566 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:57:59.655540    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f9fed8_42ab_4f2a_9069_61cd221b36ef.slice/crio-5539330d1aa01bdcb8bcd43930fcb79b3871ecd05c5e3c9d92e4f4accb49df3f WatchSource:0}: Error finding container 5539330d1aa01bdcb8bcd43930fcb79b3871ecd05c5e3c9d92e4f4accb49df3f: Status 404 returned error can't find the container with id 5539330d1aa01bdcb8bcd43930fcb79b3871ecd05c5e3c9d92e4f4accb49df3f
Nov 02 15:58:00.508689 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:00.508658    5041 generic.go:356] "Generic (PLEG): container finished" podID="c2f9fed8-42ab-4f2a-9069-61cd221b36ef" containerID="0fa0517827a3b5d6e9140cce54ba5a7c9af06555e255d353961e3ff4e5689f5d" exitCode=0
Nov 02 15:58:00.509016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:00.508741    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t" event={"ID":"c2f9fed8-42ab-4f2a-9069-61cd221b36ef","Type":"ContainerDied","Data":"0fa0517827a3b5d6e9140cce54ba5a7c9af06555e255d353961e3ff4e5689f5d"}
Nov 02 15:58:00.509016 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:00.508792    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t" event={"ID":"c2f9fed8-42ab-4f2a-9069-61cd221b36ef","Type":"ContainerStarted","Data":"5539330d1aa01bdcb8bcd43930fcb79b3871ecd05c5e3c9d92e4f4accb49df3f"}
Nov 02 15:58:01.654941 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.654913    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:58:01.802822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.802734    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg\" (UniqueName: \"kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg\") pod \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") "
Nov 02 15:58:01.802822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.802797    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca\") pod \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") "
Nov 02 15:58:01.803052 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.802854    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca\") pod \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\" (UID: \"c2f9fed8-42ab-4f2a-9069-61cd221b36ef\") "
Nov 02 15:58:01.803224 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.803201    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca" pod "c2f9fed8-42ab-4f2a-9069-61cd221b36ef" (UID: "c2f9fed8-42ab-4f2a-9069-61cd221b36ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:58:01.804784 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.804762    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca" pod "c2f9fed8-42ab-4f2a-9069-61cd221b36ef" (UID: "c2f9fed8-42ab-4f2a-9069-61cd221b36ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:58:01.804844 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.804825    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg" pod "c2f9fed8-42ab-4f2a-9069-61cd221b36ef" (UID: "c2f9fed8-42ab-4f2a-9069-61cd221b36ef"). InnerVolumeSpecName "kube-api-access-mnzqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:58:01.903824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.903793    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:01.903824 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.903824    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzqg\" (UniqueName: \"kubernetes.io/projected/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-kube-api-access-mnzqg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:01.903989 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:01.903840    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f9fed8-42ab-4f2a-9069-61cd221b36ef-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:02.521408 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:02.521357    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"
Nov 02 15:58:02.521609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:02.521402    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t" event={"ID":"c2f9fed8-42ab-4f2a-9069-61cd221b36ef","Type":"ContainerDied","Data":"5539330d1aa01bdcb8bcd43930fcb79b3871ecd05c5e3c9d92e4f4accb49df3f"}
Nov 02 15:58:02.521609 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:02.521453    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5539330d1aa01bdcb8bcd43930fcb79b3871ecd05c5e3c9d92e4f4accb49df3f"
Nov 02 15:58:06.768053 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:58:06.768015    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:58:19.199610 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.199571    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"]
Nov 02 15:58:19.200139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.200037    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c2f9fed8-42ab-4f2a-9069-61cd221b36ef" containerName="revoke-keys"
Nov 02 15:58:19.200139 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.200049    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f9fed8-42ab-4f2a-9069-61cd221b36ef" containerName="revoke-keys"
Nov 02 15:58:19.200274 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.200149    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f9fed8-42ab-4f2a-9069-61cd221b36ef" containerName="revoke-keys"
Nov 02 15:58:19.204647 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.204618    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.219264 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.219240    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:58:19.230607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.230571    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:58:19.230607 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.230594    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:58:19.230798 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.230694    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:58:19.233162 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.233121    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4pp7\" (UniqueName: \"kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.233283 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.233184    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.233390 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.233279    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.234490 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.234469    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"]
Nov 02 15:58:19.334116 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.334083    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4pp7\" (UniqueName: \"kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.334279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.334130    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.334279 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.334195    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.334830 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.334805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.336480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.336457    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.376254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.376231    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4pp7\" (UniqueName: \"kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7\") pod \"maas-api-revoke-sub-2e1f5b5f-c2bpx\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.522586 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.522537    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:19.817835 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:19.817806    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"]
Nov 02 15:58:19.819490 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:58:19.819457    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53c835e_2d87_491f_9e52_6b054ca44032.slice/crio-f934156c2c07fbb993a729f900c98723d1fbb8209aa4769859c2b983ba5dd0c6 WatchSource:0}: Error finding container f934156c2c07fbb993a729f900c98723d1fbb8209aa4769859c2b983ba5dd0c6: Status 404 returned error can't find the container with id f934156c2c07fbb993a729f900c98723d1fbb8209aa4769859c2b983ba5dd0c6
Nov 02 15:58:20.679485 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:20.679458    5041 generic.go:356] "Generic (PLEG): container finished" podID="c53c835e-2d87-491f-9e52-6b054ca44032" containerID="ce5cb2174d828b79dbd31532ffb6d3509540ca23b751ab86d0666144dc8bb476" exitCode=0
Nov 02 15:58:20.679826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:20.679539    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx" event={"ID":"c53c835e-2d87-491f-9e52-6b054ca44032","Type":"ContainerDied","Data":"ce5cb2174d828b79dbd31532ffb6d3509540ca23b751ab86d0666144dc8bb476"}
Nov 02 15:58:20.679826 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:20.679599    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx" event={"ID":"c53c835e-2d87-491f-9e52-6b054ca44032","Type":"ContainerStarted","Data":"f934156c2c07fbb993a729f900c98723d1fbb8209aa4769859c2b983ba5dd0c6"}
Nov 02 15:58:21.822662 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.822638    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:21.854208 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.854179    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca\") pod \"c53c835e-2d87-491f-9e52-6b054ca44032\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") "
Nov 02 15:58:21.854346 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.854285    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca\") pod \"c53c835e-2d87-491f-9e52-6b054ca44032\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") "
Nov 02 15:58:21.854446 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.854343    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7\" (UniqueName: \"kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7\") pod \"c53c835e-2d87-491f-9e52-6b054ca44032\" (UID: \"c53c835e-2d87-491f-9e52-6b054ca44032\") "
Nov 02 15:58:21.854670 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.854644    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca" pod "c53c835e-2d87-491f-9e52-6b054ca44032" (UID: "c53c835e-2d87-491f-9e52-6b054ca44032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:58:21.854761 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.854721    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:21.856213 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.856187    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca" pod "c53c835e-2d87-491f-9e52-6b054ca44032" (UID: "c53c835e-2d87-491f-9e52-6b054ca44032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:58:21.856314 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.856295    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7" pod "c53c835e-2d87-491f-9e52-6b054ca44032" (UID: "c53c835e-2d87-491f-9e52-6b054ca44032"). InnerVolumeSpecName "kube-api-access-l4pp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:58:21.956299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.956235    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53c835e-2d87-491f-9e52-6b054ca44032-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:21.956299 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:21.956261    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4pp7\" (UniqueName: \"kubernetes.io/projected/c53c835e-2d87-491f-9e52-6b054ca44032-kube-api-access-l4pp7\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:22.692013 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:22.691980    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx" event={"ID":"c53c835e-2d87-491f-9e52-6b054ca44032","Type":"ContainerDied","Data":"f934156c2c07fbb993a729f900c98723d1fbb8209aa4769859c2b983ba5dd0c6"}
Nov 02 15:58:22.692170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:22.692027    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f934156c2c07fbb993a729f900c98723d1fbb8209aa4769859c2b983ba5dd0c6"
Nov 02 15:58:22.692170 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:22.691994    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"
Nov 02 15:58:30.608441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.608400    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"]
Nov 02 15:58:30.609069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.609031    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c53c835e-2d87-491f-9e52-6b054ca44032" containerName="revoke-keys"
Nov 02 15:58:30.609069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.609054    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53c835e-2d87-491f-9e52-6b054ca44032" containerName="revoke-keys"
Nov 02 15:58:30.609243 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.609228    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53c835e-2d87-491f-9e52-6b054ca44032" containerName="revoke-keys"
Nov 02 15:58:30.617419 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.617339    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.634400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.634374    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"]
Nov 02 15:58:30.642885 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.642859    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:58:30.643031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.642882    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:58:30.643031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.642918    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:58:30.643143 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.643034    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:58:30.735811 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.735773    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.736004 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.735945    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.736069 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.736002    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxpg\" (UniqueName: \"kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.837510 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.837479    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.837646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.837522    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxpg\" (UniqueName: \"kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.837646 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.837576    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.838241 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.838221    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.839748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.839726    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.860399 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.860328    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxpg\" (UniqueName: \"kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg\") pod \"maas-api-revoke-sub-ec843018-ggdhn\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:30.930684 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:30.930662    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:31.252010 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:31.251973    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"]
Nov 02 15:58:31.253503 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:58:31.253467    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adba059_fb39_4fee_9e91_368c18837d73.slice/crio-d73b3bdf4ddfaaf1ab80c464cf051d40b6284974193cad27631e29f3c917fc0c WatchSource:0}: Error finding container d73b3bdf4ddfaaf1ab80c464cf051d40b6284974193cad27631e29f3c917fc0c: Status 404 returned error can't find the container with id d73b3bdf4ddfaaf1ab80c464cf051d40b6284974193cad27631e29f3c917fc0c
Nov 02 15:58:31.743416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:31.743384    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn" event={"ID":"4adba059-fb39-4fee-9e91-368c18837d73","Type":"ContainerStarted","Data":"32aa15154db1cc2d9b131978e18c5c0b8b68939145841b9558201fa973d9cbb6"}
Nov 02 15:58:31.743744 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:31.743444    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn" event={"ID":"4adba059-fb39-4fee-9e91-368c18837d73","Type":"ContainerStarted","Data":"d73b3bdf4ddfaaf1ab80c464cf051d40b6284974193cad27631e29f3c917fc0c"}
Nov 02 15:58:31.785257 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:31.785177    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn" podStartSLOduration=1.7851651560000001 podStartE2EDuration="1.785165156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:58:31.783052621 +0000 UTC m=+3764.041122070" watchObservedRunningTime="2026-11-02 15:58:31.785165156 +0000 UTC m=+3764.043234605"
Nov 02 15:58:32.750305 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:32.750269    5041 generic.go:356] "Generic (PLEG): container finished" podID="4adba059-fb39-4fee-9e91-368c18837d73" containerID="32aa15154db1cc2d9b131978e18c5c0b8b68939145841b9558201fa973d9cbb6" exitCode=0
Nov 02 15:58:32.750721 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:32.750359    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn" event={"ID":"4adba059-fb39-4fee-9e91-368c18837d73","Type":"ContainerDied","Data":"32aa15154db1cc2d9b131978e18c5c0b8b68939145841b9558201fa973d9cbb6"}
Nov 02 15:58:33.906788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.906764    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:33.967207 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.967182    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg\" (UniqueName: \"kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg\") pod \"4adba059-fb39-4fee-9e91-368c18837d73\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") "
Nov 02 15:58:33.967353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.967246    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca\") pod \"4adba059-fb39-4fee-9e91-368c18837d73\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") "
Nov 02 15:58:33.967353 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.967314    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca\") pod \"4adba059-fb39-4fee-9e91-368c18837d73\" (UID: \"4adba059-fb39-4fee-9e91-368c18837d73\") "
Nov 02 15:58:33.967706 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.967670    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca" pod "4adba059-fb39-4fee-9e91-368c18837d73" (UID: "4adba059-fb39-4fee-9e91-368c18837d73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:58:33.969244 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.969218    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg" pod "4adba059-fb39-4fee-9e91-368c18837d73" (UID: "4adba059-fb39-4fee-9e91-368c18837d73"). InnerVolumeSpecName "kube-api-access-bhxpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:58:33.969340 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:33.969281    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca" pod "4adba059-fb39-4fee-9e91-368c18837d73" (UID: "4adba059-fb39-4fee-9e91-368c18837d73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:58:34.068716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:34.068664    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxpg\" (UniqueName: \"kubernetes.io/projected/4adba059-fb39-4fee-9e91-368c18837d73-kube-api-access-bhxpg\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:34.068716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:34.068687    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:34.068716 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:34.068698    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adba059-fb39-4fee-9e91-368c18837d73-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:58:34.765056 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:34.765028    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"
Nov 02 15:58:34.765232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:34.765066    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn" event={"ID":"4adba059-fb39-4fee-9e91-368c18837d73","Type":"ContainerDied","Data":"d73b3bdf4ddfaaf1ab80c464cf051d40b6284974193cad27631e29f3c917fc0c"}
Nov 02 15:58:34.765232 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:34.765121    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d73b3bdf4ddfaaf1ab80c464cf051d40b6284974193cad27631e29f3c917fc0c"
Nov 02 15:58:36.769084 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:58:36.769035    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:58:49.482494 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:49.482453    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:58:49.483113 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:49.483074    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:58:58.977834 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:58.977800    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"]
Nov 02 15:58:58.978456 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:58.978434    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4adba059-fb39-4fee-9e91-368c18837d73" containerName="revoke-keys"
Nov 02 15:58:58.978556 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:58.978457    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adba059-fb39-4fee-9e91-368c18837d73" containerName="revoke-keys"
Nov 02 15:58:58.978637 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:58.978574    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adba059-fb39-4fee-9e91-368c18837d73" containerName="revoke-keys"
Nov 02 15:58:58.981546 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:58.981524    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:58.995748 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:58.995706    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:58:59.007121 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.007095    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:58:59.007275 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.007127    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:58:59.007341 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.007165    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:58:59.010085 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.010064    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"]
Nov 02 15:58:59.028180 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.028147    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gz9l\" (UniqueName: \"kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.028334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.028216    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.028334 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.028281    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.129476 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.129445    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gz9l\" (UniqueName: \"kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.129620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.129500    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.129620 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.129528    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.130215 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.130191    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.131817 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.131800    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.152355 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.152325    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gz9l\" (UniqueName: \"kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l\") pod \"maas-api-revoke-sub-046e1343-zx44h\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.295430 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.295354    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:58:59.595254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.595226    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"]
Nov 02 15:58:59.597106 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:58:59.597076    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc204c6af_fb7d_4198_8959_8cf16cff1883.slice/crio-59b69fe3d4052c57e71fd8883455a231a40144534374c9581b654981c608f03a WatchSource:0}: Error finding container 59b69fe3d4052c57e71fd8883455a231a40144534374c9581b654981c608f03a: Status 404 returned error can't find the container with id 59b69fe3d4052c57e71fd8883455a231a40144534374c9581b654981c608f03a
Nov 02 15:58:59.946349 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:58:59.946321    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h" event={"ID":"c204c6af-fb7d-4198-8959-8cf16cff1883","Type":"ContainerStarted","Data":"59b69fe3d4052c57e71fd8883455a231a40144534374c9581b654981c608f03a"}
Nov 02 15:59:00.954553 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:00.954521    5041 generic.go:356] "Generic (PLEG): container finished" podID="c204c6af-fb7d-4198-8959-8cf16cff1883" containerID="e72e9e4ebc4142523ec6f56c8c3b18246f7efa5ad7f3ac218e91f2bfa53faf7a" exitCode=0
Nov 02 15:59:00.954984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:00.954593    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h" event={"ID":"c204c6af-fb7d-4198-8959-8cf16cff1883","Type":"ContainerDied","Data":"e72e9e4ebc4142523ec6f56c8c3b18246f7efa5ad7f3ac218e91f2bfa53faf7a"}
Nov 02 15:59:02.100895 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.100863    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:59:02.155619 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.155592    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l\" (UniqueName: \"kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l\") pod \"c204c6af-fb7d-4198-8959-8cf16cff1883\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") "
Nov 02 15:59:02.155791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.155717    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca\") pod \"c204c6af-fb7d-4198-8959-8cf16cff1883\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") "
Nov 02 15:59:02.155791 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.155760    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca\") pod \"c204c6af-fb7d-4198-8959-8cf16cff1883\" (UID: \"c204c6af-fb7d-4198-8959-8cf16cff1883\") "
Nov 02 15:59:02.156196 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.156162    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca" pod "c204c6af-fb7d-4198-8959-8cf16cff1883" (UID: "c204c6af-fb7d-4198-8959-8cf16cff1883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:59:02.157728 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.157699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l" pod "c204c6af-fb7d-4198-8959-8cf16cff1883" (UID: "c204c6af-fb7d-4198-8959-8cf16cff1883"). InnerVolumeSpecName "kube-api-access-5gz9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:59:02.157910 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.157887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca" pod "c204c6af-fb7d-4198-8959-8cf16cff1883" (UID: "c204c6af-fb7d-4198-8959-8cf16cff1883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:59:02.257323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.257264    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:02.257323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.257289    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c204c6af-fb7d-4198-8959-8cf16cff1883-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:02.257323 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.257298    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gz9l\" (UniqueName: \"kubernetes.io/projected/c204c6af-fb7d-4198-8959-8cf16cff1883-kube-api-access-5gz9l\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:02.965697 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.965666    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"
Nov 02 15:59:02.965852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.965699    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h" event={"ID":"c204c6af-fb7d-4198-8959-8cf16cff1883","Type":"ContainerDied","Data":"59b69fe3d4052c57e71fd8883455a231a40144534374c9581b654981c608f03a"}
Nov 02 15:59:02.965852 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:02.965742    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b69fe3d4052c57e71fd8883455a231a40144534374c9581b654981c608f03a"
Nov 02 15:59:06.767175 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:59:06.767134    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:59:09.350786 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.350747    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"]
Nov 02 15:59:09.351203 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.351189    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c204c6af-fb7d-4198-8959-8cf16cff1883" containerName="revoke-keys"
Nov 02 15:59:09.351256 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.351205    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c204c6af-fb7d-4198-8959-8cf16cff1883" containerName="revoke-keys"
Nov 02 15:59:09.351296 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.351274    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c204c6af-fb7d-4198-8959-8cf16cff1883" containerName="revoke-keys"
Nov 02 15:59:09.354680 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.354664    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.378671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.378650    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:59:09.383400 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.383379    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:59:09.398017 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.397993    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:59:09.398288 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.398270    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:59:09.407530 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.407507    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"]
Nov 02 15:59:09.416193 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.416166    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.416317 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.416200    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqj5\" (UniqueName: \"kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.416436 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.416407    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.517720 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.517688    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.517853 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.517742    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.517919 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.517858    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqj5\" (UniqueName: \"kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.518356 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.518332    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.520031 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.520013    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.550049 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.550025    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqj5\" (UniqueName: \"kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5\") pod \"maas-api-revoke-sub-d3fba056-b9zzz\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.667760 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.667702    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:09.981457 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:09.981428    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"]
Nov 02 15:59:09.982587 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:59:09.982563    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362e176b_9ef8_4dfd_b1f8_16f7e68c8c3b.slice/crio-aa09a55869eca8ffce6cb7e67aecf3dcec61f0a2cd204c32829c403fa5ccc289 WatchSource:0}: Error finding container aa09a55869eca8ffce6cb7e67aecf3dcec61f0a2cd204c32829c403fa5ccc289: Status 404 returned error can't find the container with id aa09a55869eca8ffce6cb7e67aecf3dcec61f0a2cd204c32829c403fa5ccc289
Nov 02 15:59:10.019512 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:10.019484    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz" event={"ID":"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b","Type":"ContainerStarted","Data":"aa09a55869eca8ffce6cb7e67aecf3dcec61f0a2cd204c32829c403fa5ccc289"}
Nov 02 15:59:11.031287 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:11.031190    5041 generic.go:356] "Generic (PLEG): container finished" podID="362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" containerID="b2b0852174e5b73820b497bf055b6c2dc763c607f48a2e1dd8aac73f2c5fa03e" exitCode=0
Nov 02 15:59:11.031804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:11.031273    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz" event={"ID":"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b","Type":"ContainerDied","Data":"b2b0852174e5b73820b497bf055b6c2dc763c607f48a2e1dd8aac73f2c5fa03e"}
Nov 02 15:59:12.690396 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.683516    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:12.764734 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.764699    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5\" (UniqueName: \"kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5\") pod \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") "
Nov 02 15:59:12.765480 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.765454    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca\") pod \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") "
Nov 02 15:59:12.765704 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.765684    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca\") pod \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\" (UID: \"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b\") "
Nov 02 15:59:12.768554 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.768526    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca" pod "362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" (UID: "362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:59:12.787384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.777515    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5" pod "362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" (UID: "362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b"). InnerVolumeSpecName "kube-api-access-ttqj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:59:12.787384 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.778458    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca" pod "362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" (UID: "362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:59:12.908168 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.908081    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:12.908416 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.908395    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:12.908626 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:12.908606    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttqj5\" (UniqueName: \"kubernetes.io/projected/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b-kube-api-access-ttqj5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:13.103388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:13.102376    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz" event={"ID":"362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b","Type":"ContainerDied","Data":"aa09a55869eca8ffce6cb7e67aecf3dcec61f0a2cd204c32829c403fa5ccc289"}
Nov 02 15:59:13.103388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:13.102442    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa09a55869eca8ffce6cb7e67aecf3dcec61f0a2cd204c32829c403fa5ccc289"
Nov 02 15:59:13.103388 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:13.102512    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"
Nov 02 15:59:35.797822 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.797743    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"]
Nov 02 15:59:35.798295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.798182    5041 cpu_manager.go:414] "containerMap: removing container" podUID="362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" containerName="revoke-keys"
Nov 02 15:59:35.798295 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.798196    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" containerName="revoke-keys"
Nov 02 15:59:35.798392 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.798296    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" containerName="revoke-keys"
Nov 02 15:59:35.801804 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.801782    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:35.827352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.827305    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:59:35.827352 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.827305    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:59:35.827519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.827373    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:59:35.827519 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.827394    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:59:35.831643 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.831616    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"]
Nov 02 15:59:35.976043 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.976011    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clx7v\" (UniqueName: \"kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:35.976200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.976076    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:35.976200 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:35.976152    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.076984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.076919    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clx7v\" (UniqueName: \"kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.076984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.076957    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.076984 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.076979    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.077603 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.077578    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.079276 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.079254    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.100015 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.099991    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clx7v\" (UniqueName: \"kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v\") pod \"maas-api-revoke-sub-ae594979-mzsfh\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.115765 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.115746    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:36.417289 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:36.417257    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"]
Nov 02 15:59:36.418590 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:59:36.418565    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7943581_ce7c_4aaa_a314_c9a9e8e2a1a4.slice/crio-ebd015d2f0538aae5ff2018521c61a469f9538c06a0f8885933fbe9721f3d987 WatchSource:0}: Error finding container ebd015d2f0538aae5ff2018521c61a469f9538c06a0f8885933fbe9721f3d987: Status 404 returned error can't find the container with id ebd015d2f0538aae5ff2018521c61a469f9538c06a0f8885933fbe9721f3d987
Nov 02 15:59:36.766322 ip-10-0-133-143 kubenswrapper[5041]: E1102 15:59:36.766287    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 15:59:37.277081 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:37.277048    5041 generic.go:356] "Generic (PLEG): container finished" podID="a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" containerID="8cfe6f2265a2b4b3099641022afe63d78b9c6d6d7f5d592d048991f1ac374646" exitCode=0
Nov 02 15:59:37.277415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:37.277131    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh" event={"ID":"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4","Type":"ContainerDied","Data":"8cfe6f2265a2b4b3099641022afe63d78b9c6d6d7f5d592d048991f1ac374646"}
Nov 02 15:59:37.277415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:37.277180    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh" event={"ID":"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4","Type":"ContainerStarted","Data":"ebd015d2f0538aae5ff2018521c61a469f9538c06a0f8885933fbe9721f3d987"}
Nov 02 15:59:38.417774 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.417751    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:38.497742 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.497693    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v\" (UniqueName: \"kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v\") pod \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") "
Nov 02 15:59:38.499671 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.499644    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v" pod "a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" (UID: "a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4"). InnerVolumeSpecName "kube-api-access-clx7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:59:38.598602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.598546    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca\") pod \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") "
Nov 02 15:59:38.598602 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.598584    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca\") pod \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\" (UID: \"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4\") "
Nov 02 15:59:38.598860 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.598843    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clx7v\" (UniqueName: \"kubernetes.io/projected/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-kube-api-access-clx7v\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:38.598935 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.598909    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca" pod "a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" (UID: "a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:59:38.600562 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.600543    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca" pod "a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" (UID: "a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:59:38.699551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.699525    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:38.699551 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:38.699547    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 15:59:39.290078 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:39.290051    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"
Nov 02 15:59:39.290216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:39.290082    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh" event={"ID":"a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4","Type":"ContainerDied","Data":"ebd015d2f0538aae5ff2018521c61a469f9538c06a0f8885933fbe9721f3d987"}
Nov 02 15:59:39.290216 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:39.290137    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd015d2f0538aae5ff2018521c61a469f9538c06a0f8885933fbe9721f3d987"
Nov 02 15:59:57.629301 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.629259    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"]
Nov 02 15:59:57.629861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.629745    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" containerName="revoke-keys"
Nov 02 15:59:57.629861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.629758    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" containerName="revoke-keys"
Nov 02 15:59:57.629861 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.629836    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" containerName="revoke-keys"
Nov 02 15:59:57.633147 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.633127    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.647437 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.647407    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:59:57.658651 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.658623    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:59:57.659415 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.659391    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:59:57.659753 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.659730    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:59:57.663105 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.663076    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"]
Nov 02 15:59:57.708585 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.708560    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pd9l\" (UniqueName: \"kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.708715 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.708616    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.708762 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.708742    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.809657 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.809630    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.809788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.809688    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pd9l\" (UniqueName: \"kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.809788 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.809731    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.810441 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.810419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.812102 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.812078    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.832112 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.832088    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pd9l\" (UniqueName: \"kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l\") pod \"maas-api-revoke-sub-a4aef7d7-r7mt6\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:57.946374 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:57.946330    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 15:59:58.246174 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:58.246143    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"]
Nov 02 15:59:58.247227 ip-10-0-133-143 kubenswrapper[5041]: W1102 15:59:58.247199    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88e3d74_37ff_4c3f_a1c9_b4c51376c877.slice/crio-5b6e7f6ce3348c834a713213dd8bb6f1144e5ec244bc898b9d240ca5fb33259b WatchSource:0}: Error finding container 5b6e7f6ce3348c834a713213dd8bb6f1144e5ec244bc898b9d240ca5fb33259b: Status 404 returned error can't find the container with id 5b6e7f6ce3348c834a713213dd8bb6f1144e5ec244bc898b9d240ca5fb33259b
Nov 02 15:59:58.422254 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:58.422226    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6" event={"ID":"d88e3d74-37ff-4c3f-a1c9-b4c51376c877","Type":"ContainerStarted","Data":"5b6e7f6ce3348c834a713213dd8bb6f1144e5ec244bc898b9d240ca5fb33259b"}
Nov 02 15:59:59.429266 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:59.429235    5041 generic.go:356] "Generic (PLEG): container finished" podID="d88e3d74-37ff-4c3f-a1c9-b4c51376c877" containerID="8d427649a48b7c9f848246ba965889712b50ae3ac1b3c3b367e849f513f66cc9" exitCode=0
Nov 02 15:59:59.429633 ip-10-0-133-143 kubenswrapper[5041]: I1102 15:59:59.429317    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6" event={"ID":"d88e3d74-37ff-4c3f-a1c9-b4c51376c877","Type":"ContainerDied","Data":"8d427649a48b7c9f848246ba965889712b50ae3ac1b3c3b367e849f513f66cc9"}
Nov 02 16:00:00.570309 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.570280    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 16:00:00.735471 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.735407    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca\") pod \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") "
Nov 02 16:00:00.735471 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.735455    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l\" (UniqueName: \"kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l\") pod \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") "
Nov 02 16:00:00.735634 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.735580    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca\") pod \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\" (UID: \"d88e3d74-37ff-4c3f-a1c9-b4c51376c877\") "
Nov 02 16:00:00.735803 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.735782    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca" pod "d88e3d74-37ff-4c3f-a1c9-b4c51376c877" (UID: "d88e3d74-37ff-4c3f-a1c9-b4c51376c877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:00:00.735974 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.735950    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:00.737682 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.737651    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca" pod "d88e3d74-37ff-4c3f-a1c9-b4c51376c877" (UID: "d88e3d74-37ff-4c3f-a1c9-b4c51376c877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:00:00.737682 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.737659    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l" pod "d88e3d74-37ff-4c3f-a1c9-b4c51376c877" (UID: "d88e3d74-37ff-4c3f-a1c9-b4c51376c877"). InnerVolumeSpecName "kube-api-access-4pd9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:00:00.836786 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.836756    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:00.836786 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:00.836783    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pd9l\" (UniqueName: \"kubernetes.io/projected/d88e3d74-37ff-4c3f-a1c9-b4c51376c877-kube-api-access-4pd9l\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:01.443111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:01.443082    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"
Nov 02 16:00:01.443270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:01.443114    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6" event={"ID":"d88e3d74-37ff-4c3f-a1c9-b4c51376c877","Type":"ContainerDied","Data":"5b6e7f6ce3348c834a713213dd8bb6f1144e5ec244bc898b9d240ca5fb33259b"}
Nov 02 16:00:01.443270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:01.443165    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b6e7f6ce3348c834a713213dd8bb6f1144e5ec244bc898b9d240ca5fb33259b"
Nov 02 16:00:06.767547 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:00:06.767508    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:00:17.845824 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.845786    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"]
Nov 02 16:00:17.846554 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.846523    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d88e3d74-37ff-4c3f-a1c9-b4c51376c877" containerName="revoke-keys"
Nov 02 16:00:17.846554 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.846552    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88e3d74-37ff-4c3f-a1c9-b4c51376c877" containerName="revoke-keys"
Nov 02 16:00:17.846742 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.846698    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88e3d74-37ff-4c3f-a1c9-b4c51376c877" containerName="revoke-keys"
Nov 02 16:00:17.850247 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.850230    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:17.864926 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.864904    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:00:17.875891 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.875861    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:00:17.875891 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.875868    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:00:17.876081 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.876022    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:00:17.879106 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.879081    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"]
Nov 02 16:00:17.933218 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.933188    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72p4w\" (UniqueName: \"kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:17.933399 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.933250    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:17.933445 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:17.933384    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.034046 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.034008    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.034237 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.034098    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.034237 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.034153    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72p4w\" (UniqueName: \"kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.034807 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.034775    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.036451 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.036432    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.056532 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.056504    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72p4w\" (UniqueName: \"kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w\") pod \"maas-api-revoke-sub-79805ca9-jvnwj\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.163948 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.163875    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:18.462555 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.462525    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"]
Nov 02 16:00:18.464285 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:00:18.464254    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ca2414_9735_4d0e_8458_0ebb43f7640e.slice/crio-73336fa961d6d3ff0449fea2176238cf4a6e879a776ff0a0de844f62440f522f WatchSource:0}: Error finding container 73336fa961d6d3ff0449fea2176238cf4a6e879a776ff0a0de844f62440f522f: Status 404 returned error can't find the container with id 73336fa961d6d3ff0449fea2176238cf4a6e879a776ff0a0de844f62440f522f
Nov 02 16:00:18.587262 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:18.587228    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj" event={"ID":"44ca2414-9735-4d0e-8458-0ebb43f7640e","Type":"ContainerStarted","Data":"73336fa961d6d3ff0449fea2176238cf4a6e879a776ff0a0de844f62440f522f"}
Nov 02 16:00:19.594952 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:19.594865    5041 generic.go:356] "Generic (PLEG): container finished" podID="44ca2414-9735-4d0e-8458-0ebb43f7640e" containerID="59a1461105e6d89b2db957ba75d33737c9d4008e678b3aae2c6fc6adb9e4f1bf" exitCode=0
Nov 02 16:00:19.594952 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:19.594927    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj" event={"ID":"44ca2414-9735-4d0e-8458-0ebb43f7640e","Type":"ContainerDied","Data":"59a1461105e6d89b2db957ba75d33737c9d4008e678b3aae2c6fc6adb9e4f1bf"}
Nov 02 16:00:20.750224 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.750190    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:20.859768 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.859683    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca\") pod \"44ca2414-9735-4d0e-8458-0ebb43f7640e\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") "
Nov 02 16:00:20.859950 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.859775    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w\" (UniqueName: \"kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w\") pod \"44ca2414-9735-4d0e-8458-0ebb43f7640e\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") "
Nov 02 16:00:20.859950 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.859814    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca\") pod \"44ca2414-9735-4d0e-8458-0ebb43f7640e\" (UID: \"44ca2414-9735-4d0e-8458-0ebb43f7640e\") "
Nov 02 16:00:20.860188 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.860165    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca" pod "44ca2414-9735-4d0e-8458-0ebb43f7640e" (UID: "44ca2414-9735-4d0e-8458-0ebb43f7640e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:00:20.861804 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.861781    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w" pod "44ca2414-9735-4d0e-8458-0ebb43f7640e" (UID: "44ca2414-9735-4d0e-8458-0ebb43f7640e"). InnerVolumeSpecName "kube-api-access-72p4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:00:20.861804 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.861793    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca" pod "44ca2414-9735-4d0e-8458-0ebb43f7640e" (UID: "44ca2414-9735-4d0e-8458-0ebb43f7640e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:00:20.961201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.961168    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72p4w\" (UniqueName: \"kubernetes.io/projected/44ca2414-9735-4d0e-8458-0ebb43f7640e-kube-api-access-72p4w\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:20.961201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.961197    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:20.961201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:20.961208    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ca2414-9735-4d0e-8458-0ebb43f7640e-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:21.607045 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:21.607009    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"
Nov 02 16:00:21.607045 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:21.607037    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj" event={"ID":"44ca2414-9735-4d0e-8458-0ebb43f7640e","Type":"ContainerDied","Data":"73336fa961d6d3ff0449fea2176238cf4a6e879a776ff0a0de844f62440f522f"}
Nov 02 16:00:21.607281 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:21.607086    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73336fa961d6d3ff0449fea2176238cf4a6e879a776ff0a0de844f62440f522f"
Nov 02 16:00:32.032826 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:32.032784    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"]
Nov 02 16:00:32.042924 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:32.042891    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfde0b7-nzbrf"]
Nov 02 16:00:33.908387 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:33.900185    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f01110d-87a9-467f-95be-4c4195a78793" path="/var/lib/kubelet/pods/5f01110d-87a9-467f-95be-4c4195a78793/volumes"
Nov 02 16:00:34.500386 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:34.499691    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:00:34.500386 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:34.499769    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:00:36.767879 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:00:36.767840    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:00:39.183508 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.183470    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"]
Nov 02 16:00:39.184041 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.183944    5041 cpu_manager.go:414] "containerMap: removing container" podUID="44ca2414-9735-4d0e-8458-0ebb43f7640e" containerName="revoke-keys"
Nov 02 16:00:39.184041 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.183957    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ca2414-9735-4d0e-8458-0ebb43f7640e" containerName="revoke-keys"
Nov 02 16:00:39.184041 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.184037    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ca2414-9735-4d0e-8458-0ebb43f7640e" containerName="revoke-keys"
Nov 02 16:00:39.187309 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.187279    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.217840 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.217814    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:00:39.217990 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.217830    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:00:39.217990 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.217859    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:00:39.218071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.217987    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:00:39.232597 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.232573    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"]
Nov 02 16:00:39.266591 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.266559    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.266746 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.266697    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzvz\" (UniqueName: \"kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.266746 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.266733    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.367524 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.367493    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.367684 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.367561    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzvz\" (UniqueName: \"kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.367684 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.367588    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.368243 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.368215    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.369873 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.369847    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.390465 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.390438    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzvz\" (UniqueName: \"kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz\") pod \"maas-api-revoke-sub-0b7e8edb-ll7pw\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.500153 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.500078    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:39.802801 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:39.802765    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"]
Nov 02 16:00:39.804197 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:00:39.804173    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68919812_bfc7_4ff4_a406_bca192b71070.slice/crio-b4289087efe82297d692bcd1098cf808b528a4310d27ce4e60aae251bce90401 WatchSource:0}: Error finding container b4289087efe82297d692bcd1098cf808b528a4310d27ce4e60aae251bce90401: Status 404 returned error can't find the container with id b4289087efe82297d692bcd1098cf808b528a4310d27ce4e60aae251bce90401
Nov 02 16:00:40.750427 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:40.750394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw" event={"ID":"68919812-bfc7-4ff4-a406-bca192b71070","Type":"ContainerStarted","Data":"62dbabe8990ff435431d7906d228df80fdb98733726293d21267a501d77b1750"}
Nov 02 16:00:40.750878 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:40.750484    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw" event={"ID":"68919812-bfc7-4ff4-a406-bca192b71070","Type":"ContainerStarted","Data":"b4289087efe82297d692bcd1098cf808b528a4310d27ce4e60aae251bce90401"}
Nov 02 16:00:40.792400 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:40.792334    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw" podStartSLOduration=1.792320742 podStartE2EDuration="1.792320742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:40.789288708 +0000 UTC m=+3893.047358159" watchObservedRunningTime="2026-11-02 16:00:40.792320742 +0000 UTC m=+3893.050390191"
Nov 02 16:00:41.758411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:41.758346    5041 generic.go:356] "Generic (PLEG): container finished" podID="68919812-bfc7-4ff4-a406-bca192b71070" containerID="62dbabe8990ff435431d7906d228df80fdb98733726293d21267a501d77b1750" exitCode=0
Nov 02 16:00:41.758821 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:41.758427    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw" event={"ID":"68919812-bfc7-4ff4-a406-bca192b71070","Type":"ContainerDied","Data":"62dbabe8990ff435431d7906d228df80fdb98733726293d21267a501d77b1750"}
Nov 02 16:00:42.914926 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:42.914902    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:43.004771 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.004735    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz\" (UniqueName: \"kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz\") pod \"68919812-bfc7-4ff4-a406-bca192b71070\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") "
Nov 02 16:00:43.004949 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.004819    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca\") pod \"68919812-bfc7-4ff4-a406-bca192b71070\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") "
Nov 02 16:00:43.004949 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.004935    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca\") pod \"68919812-bfc7-4ff4-a406-bca192b71070\" (UID: \"68919812-bfc7-4ff4-a406-bca192b71070\") "
Nov 02 16:00:43.005181 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.005158    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca" pod "68919812-bfc7-4ff4-a406-bca192b71070" (UID: "68919812-bfc7-4ff4-a406-bca192b71070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:00:43.005290 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.005272    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:43.006928 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.006908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz" pod "68919812-bfc7-4ff4-a406-bca192b71070" (UID: "68919812-bfc7-4ff4-a406-bca192b71070"). InnerVolumeSpecName "kube-api-access-6tzvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:00:43.007014 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.006908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca" pod "68919812-bfc7-4ff4-a406-bca192b71070" (UID: "68919812-bfc7-4ff4-a406-bca192b71070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:00:43.106728 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.106648    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tzvz\" (UniqueName: \"kubernetes.io/projected/68919812-bfc7-4ff4-a406-bca192b71070-kube-api-access-6tzvz\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:43.106728 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.106681    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68919812-bfc7-4ff4-a406-bca192b71070-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:43.771640 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.771600    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw" event={"ID":"68919812-bfc7-4ff4-a406-bca192b71070","Type":"ContainerDied","Data":"b4289087efe82297d692bcd1098cf808b528a4310d27ce4e60aae251bce90401"}
Nov 02 16:00:43.771824 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.771646    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"
Nov 02 16:00:43.771824 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:43.771661    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4289087efe82297d692bcd1098cf808b528a4310d27ce4e60aae251bce90401"
Nov 02 16:00:49.400701 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:49.400589    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:00:49.421018 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:49.413456    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:00:50.436937 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.436895    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"]
Nov 02 16:00:50.437356 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.437336    5041 cpu_manager.go:414] "containerMap: removing container" podUID="68919812-bfc7-4ff4-a406-bca192b71070" containerName="revoke-keys"
Nov 02 16:00:50.437356 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.437347    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68919812-bfc7-4ff4-a406-bca192b71070" containerName="revoke-keys"
Nov 02 16:00:50.437477 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.437428    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="68919812-bfc7-4ff4-a406-bca192b71070" containerName="revoke-keys"
Nov 02 16:00:50.440697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.440679    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.455346 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.455322    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:00:50.455502 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.455486    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:00:50.468484 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.468459    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:00:50.468677 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.468522    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:00:50.471680 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.471645    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"]
Nov 02 16:00:50.578260 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.578225    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.578469 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.578263    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.578469 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.578415    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b65tr\" (UniqueName: \"kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.679575 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.679533    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.679779 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.679581    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.679779 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.679628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b65tr\" (UniqueName: \"kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.680294 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.680266    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.682020 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.681989    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.701978 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.701915    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b65tr\" (UniqueName: \"kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr\") pod \"maas-api-revoke-sub-3d4156f1-pjj79\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:50.753979 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:50.753955    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:51.054313 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:51.054279    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"]
Nov 02 16:00:51.055659 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:00:51.055632    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c33300_ffa0_40fb_ab57_483348cc0ca0.slice/crio-876d3c3d60e249634be447440f9cd265f65bd3f0ae893fd47bcdee1968f9298c WatchSource:0}: Error finding container 876d3c3d60e249634be447440f9cd265f65bd3f0ae893fd47bcdee1968f9298c: Status 404 returned error can't find the container with id 876d3c3d60e249634be447440f9cd265f65bd3f0ae893fd47bcdee1968f9298c
Nov 02 16:00:51.827160 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:51.827122    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79" event={"ID":"42c33300-ffa0-40fb-ab57-483348cc0ca0","Type":"ContainerStarted","Data":"06c14f2e799828e000c168644efd1704fd89f2ada7eec4eca8c1fb4d0e423d5e"}
Nov 02 16:00:51.827587 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:51.827185    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79" event={"ID":"42c33300-ffa0-40fb-ab57-483348cc0ca0","Type":"ContainerStarted","Data":"876d3c3d60e249634be447440f9cd265f65bd3f0ae893fd47bcdee1968f9298c"}
Nov 02 16:00:51.868736 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:51.868684    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79" podStartSLOduration=1.868668175 podStartE2EDuration="1.868668175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:51.867193883 +0000 UTC m=+3904.125263334" watchObservedRunningTime="2026-11-02 16:00:51.868668175 +0000 UTC m=+3904.126737672"
Nov 02 16:00:52.614245 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:52.614202    5041 scope.go:122] "RemoveContainer" containerID="171d25886a9d9dff6bad8b94a538406affb331ff0a891976ac492d34e3182f72"
Nov 02 16:00:52.835723 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:52.835678    5041 generic.go:356] "Generic (PLEG): container finished" podID="42c33300-ffa0-40fb-ab57-483348cc0ca0" containerID="06c14f2e799828e000c168644efd1704fd89f2ada7eec4eca8c1fb4d0e423d5e" exitCode=0
Nov 02 16:00:52.836103 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:52.835725    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79" event={"ID":"42c33300-ffa0-40fb-ab57-483348cc0ca0","Type":"ContainerDied","Data":"06c14f2e799828e000c168644efd1704fd89f2ada7eec4eca8c1fb4d0e423d5e"}
Nov 02 16:00:53.984658 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:53.984634    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:54.110868 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.110794    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca\") pod \"42c33300-ffa0-40fb-ab57-483348cc0ca0\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") "
Nov 02 16:00:54.110868 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.110844    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca\") pod \"42c33300-ffa0-40fb-ab57-483348cc0ca0\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") "
Nov 02 16:00:54.111047 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.110872    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr\" (UniqueName: \"kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr\") pod \"42c33300-ffa0-40fb-ab57-483348cc0ca0\" (UID: \"42c33300-ffa0-40fb-ab57-483348cc0ca0\") "
Nov 02 16:00:54.111190 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.111161    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca" pod "42c33300-ffa0-40fb-ab57-483348cc0ca0" (UID: "42c33300-ffa0-40fb-ab57-483348cc0ca0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:00:54.112949 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.112926    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca" pod "42c33300-ffa0-40fb-ab57-483348cc0ca0" (UID: "42c33300-ffa0-40fb-ab57-483348cc0ca0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:00:54.113040 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.113015    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr" pod "42c33300-ffa0-40fb-ab57-483348cc0ca0" (UID: "42c33300-ffa0-40fb-ab57-483348cc0ca0"). InnerVolumeSpecName "kube-api-access-b65tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:00:54.212554 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.212526    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:54.212554 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.212553    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c33300-ffa0-40fb-ab57-483348cc0ca0-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:54.212729 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.212563    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b65tr\" (UniqueName: \"kubernetes.io/projected/42c33300-ffa0-40fb-ab57-483348cc0ca0-kube-api-access-b65tr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:00:54.849988 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.849954    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"
Nov 02 16:00:54.850202 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.849954    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79" event={"ID":"42c33300-ffa0-40fb-ab57-483348cc0ca0","Type":"ContainerDied","Data":"876d3c3d60e249634be447440f9cd265f65bd3f0ae893fd47bcdee1968f9298c"}
Nov 02 16:00:54.850202 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:00:54.850087    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876d3c3d60e249634be447440f9cd265f65bd3f0ae893fd47bcdee1968f9298c"
Nov 02 16:01:04.479450 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:04.478489    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:01:04.479450 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:04.478595    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:01:06.936671 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:01:06.936607    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:01:12.956072 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:12.956032    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"]
Nov 02 16:01:12.967257 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:12.967230    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pjj79"]
Nov 02 16:01:13.006770 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.006738    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"]
Nov 02 16:01:13.007190 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.007178    5041 cpu_manager.go:414] "containerMap: removing container" podUID="42c33300-ffa0-40fb-ab57-483348cc0ca0" containerName="revoke-keys"
Nov 02 16:01:13.007245 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.007191    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c33300-ffa0-40fb-ab57-483348cc0ca0" containerName="revoke-keys"
Nov 02 16:01:13.007286 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.007263    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c33300-ffa0-40fb-ab57-483348cc0ca0" containerName="revoke-keys"
Nov 02 16:01:13.010753 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.010729    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.025838 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.025810    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:01:13.025980 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.025813    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:01:13.025980 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.025909    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:01:13.025980 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.025924    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:01:13.032597 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.032571    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"]
Nov 02 16:01:13.083670 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.083642    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz79\" (UniqueName: \"kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.083814 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.083699    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.083814 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.083791    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.185065 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.185033    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.185214 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.185079    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.185214 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.185152    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz79\" (UniqueName: \"kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.191796 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.186195    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.196820 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.196651    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.208500 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.208441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz79\" (UniqueName: \"kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79\") pod \"maas-api-revoke-sub-3d4156f1-9xcjc\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.324191 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.324164    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:13.624732 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.624698    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"]
Nov 02 16:01:13.625927 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:01:13.625903    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2d5094_c50c_453a_af22_68b79f1588dd.slice/crio-ce670fafbe07db338df0ed2634690a9958222002be3e77a30aa28e94313ad5f5 WatchSource:0}: Error finding container ce670fafbe07db338df0ed2634690a9958222002be3e77a30aa28e94313ad5f5: Status 404 returned error can't find the container with id ce670fafbe07db338df0ed2634690a9958222002be3e77a30aa28e94313ad5f5
Nov 02 16:01:13.887635 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:13.887571    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c33300-ffa0-40fb-ab57-483348cc0ca0" path="/var/lib/kubelet/pods/42c33300-ffa0-40fb-ab57-483348cc0ca0/volumes"
Nov 02 16:01:14.002423 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:14.002394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc" event={"ID":"bd2d5094-c50c-453a-af22-68b79f1588dd","Type":"ContainerStarted","Data":"6cba6a115429cb914ad12082eaa439f30e2b0f2137a1383b5657bb445928f68a"}
Nov 02 16:01:14.002776 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:14.002449    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc" event={"ID":"bd2d5094-c50c-453a-af22-68b79f1588dd","Type":"ContainerStarted","Data":"ce670fafbe07db338df0ed2634690a9958222002be3e77a30aa28e94313ad5f5"}
Nov 02 16:01:14.043670 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:14.043629    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc" podStartSLOduration=2.04361606 podStartE2EDuration="2.04361606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:14.04166604 +0000 UTC m=+3926.299735491" watchObservedRunningTime="2026-11-02 16:01:14.04361606 +0000 UTC m=+3926.301685511"
Nov 02 16:01:15.010618 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:15.010581    5041 generic.go:356] "Generic (PLEG): container finished" podID="bd2d5094-c50c-453a-af22-68b79f1588dd" containerID="6cba6a115429cb914ad12082eaa439f30e2b0f2137a1383b5657bb445928f68a" exitCode=0
Nov 02 16:01:15.011027 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:15.010610    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc" event={"ID":"bd2d5094-c50c-453a-af22-68b79f1588dd","Type":"ContainerDied","Data":"6cba6a115429cb914ad12082eaa439f30e2b0f2137a1383b5657bb445928f68a"}
Nov 02 16:01:16.168802 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.168769    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:16.316898 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.316815    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca\") pod \"bd2d5094-c50c-453a-af22-68b79f1588dd\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") "
Nov 02 16:01:16.317085 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.316951    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79\" (UniqueName: \"kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79\") pod \"bd2d5094-c50c-453a-af22-68b79f1588dd\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") "
Nov 02 16:01:16.317085 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.317012    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca\") pod \"bd2d5094-c50c-453a-af22-68b79f1588dd\" (UID: \"bd2d5094-c50c-453a-af22-68b79f1588dd\") "
Nov 02 16:01:16.317423 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.317392    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca" pod "bd2d5094-c50c-453a-af22-68b79f1588dd" (UID: "bd2d5094-c50c-453a-af22-68b79f1588dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:01:16.319007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.318982    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79" pod "bd2d5094-c50c-453a-af22-68b79f1588dd" (UID: "bd2d5094-c50c-453a-af22-68b79f1588dd"). InnerVolumeSpecName "kube-api-access-7zz79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:01:16.319112 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.319001    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca" pod "bd2d5094-c50c-453a-af22-68b79f1588dd" (UID: "bd2d5094-c50c-453a-af22-68b79f1588dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:01:16.418251 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.418219    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:01:16.418359 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.418254    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zz79\" (UniqueName: \"kubernetes.io/projected/bd2d5094-c50c-453a-af22-68b79f1588dd-kube-api-access-7zz79\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:01:16.418359 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:16.418268    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d5094-c50c-453a-af22-68b79f1588dd-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:01:17.026613 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:17.026580    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"
Nov 02 16:01:17.026812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:17.026576    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc" event={"ID":"bd2d5094-c50c-453a-af22-68b79f1588dd","Type":"ContainerDied","Data":"ce670fafbe07db338df0ed2634690a9958222002be3e77a30aa28e94313ad5f5"}
Nov 02 16:01:17.026812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:17.026744    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce670fafbe07db338df0ed2634690a9958222002be3e77a30aa28e94313ad5f5"
Nov 02 16:01:36.767269 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:01:36.767226    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:01:49.229212 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.229176    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"]
Nov 02 16:01:49.229712 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.229596    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bd2d5094-c50c-453a-af22-68b79f1588dd" containerName="revoke-keys"
Nov 02 16:01:49.229712 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.229608    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2d5094-c50c-453a-af22-68b79f1588dd" containerName="revoke-keys"
Nov 02 16:01:49.229712 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.229709    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2d5094-c50c-453a-af22-68b79f1588dd" containerName="revoke-keys"
Nov 02 16:01:49.233051 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.233032    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.255133 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.255100    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"]
Nov 02 16:01:49.263179 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.263155    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:01:49.263352 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.263166    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:01:49.263352 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.263211    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:01:49.263352 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.263278    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:01:49.351976 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.351949    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcqq\" (UniqueName: \"kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.352153 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.351989    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.352153 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.352009    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.452638 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.452597    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcqq\" (UniqueName: \"kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.452774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.452651    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.452774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.452672    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.453254 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.453235    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.455053 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.455029    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.475342 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.475318    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcqq\" (UniqueName: \"kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq\") pod \"maas-api-revoke-sub-e2fc9c5b-b2rvt\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.550661 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.550609    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:49.852214 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:49.852190    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"]
Nov 02 16:01:49.854154 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:01:49.854131    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101cfa8d_0553_4180_94e7_c9ba2ca41497.slice/crio-ace1669a1e852256931b86e058378e02045526bcc2e6f26f041c802664b24a51 WatchSource:0}: Error finding container ace1669a1e852256931b86e058378e02045526bcc2e6f26f041c802664b24a51: Status 404 returned error can't find the container with id ace1669a1e852256931b86e058378e02045526bcc2e6f26f041c802664b24a51
Nov 02 16:01:50.254169 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:50.254130    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt" event={"ID":"101cfa8d-0553-4180-94e7-c9ba2ca41497","Type":"ContainerStarted","Data":"15b6aae0caa0391a35f53e1e4b1633e3ea8c14905f78fb1bb89261bcef167a47"}
Nov 02 16:01:50.254688 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:50.254183    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt" event={"ID":"101cfa8d-0553-4180-94e7-c9ba2ca41497","Type":"ContainerStarted","Data":"ace1669a1e852256931b86e058378e02045526bcc2e6f26f041c802664b24a51"}
Nov 02 16:01:50.294688 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:50.294641    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt" podStartSLOduration=1.294626551 podStartE2EDuration="1.294626551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:50.293494051 +0000 UTC m=+3962.551563500" watchObservedRunningTime="2026-11-02 16:01:50.294626551 +0000 UTC m=+3962.552696005"
Nov 02 16:01:51.261903 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:51.261868    5041 generic.go:356] "Generic (PLEG): container finished" podID="101cfa8d-0553-4180-94e7-c9ba2ca41497" containerID="15b6aae0caa0391a35f53e1e4b1633e3ea8c14905f78fb1bb89261bcef167a47" exitCode=0
Nov 02 16:01:51.262270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:51.261951    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt" event={"ID":"101cfa8d-0553-4180-94e7-c9ba2ca41497","Type":"ContainerDied","Data":"15b6aae0caa0391a35f53e1e4b1633e3ea8c14905f78fb1bb89261bcef167a47"}
Nov 02 16:01:52.412724 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.412697    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:52.483320 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.483285    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq\" (UniqueName: \"kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq\") pod \"101cfa8d-0553-4180-94e7-c9ba2ca41497\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") "
Nov 02 16:01:52.483527 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.483341    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca\") pod \"101cfa8d-0553-4180-94e7-c9ba2ca41497\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") "
Nov 02 16:01:52.483527 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.483496    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca\") pod \"101cfa8d-0553-4180-94e7-c9ba2ca41497\" (UID: \"101cfa8d-0553-4180-94e7-c9ba2ca41497\") "
Nov 02 16:01:52.483899 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.483864    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca" pod "101cfa8d-0553-4180-94e7-c9ba2ca41497" (UID: "101cfa8d-0553-4180-94e7-c9ba2ca41497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:01:52.485508 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.485481    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca" pod "101cfa8d-0553-4180-94e7-c9ba2ca41497" (UID: "101cfa8d-0553-4180-94e7-c9ba2ca41497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:01:52.485650 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.485614    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq" pod "101cfa8d-0553-4180-94e7-c9ba2ca41497" (UID: "101cfa8d-0553-4180-94e7-c9ba2ca41497"). InnerVolumeSpecName "kube-api-access-mfcqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:01:52.584844 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.584763    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfcqq\" (UniqueName: \"kubernetes.io/projected/101cfa8d-0553-4180-94e7-c9ba2ca41497-kube-api-access-mfcqq\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:01:52.584844 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.584794    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:01:52.584844 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.584809    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cfa8d-0553-4180-94e7-c9ba2ca41497-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:01:52.639203 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.639171    5041 scope.go:122] "RemoveContainer" containerID="f16336d4697c249a4d14e9e48565d7a88a5b2791fe73f724d66bd4b24f5516e0"
Nov 02 16:01:52.650452 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:52.650426    5041 scope.go:122] "RemoveContainer" containerID="de35982f08b53e8a0a4186dd61e000f7f41b1a5bcb866560ee19d98fed7e0117"
Nov 02 16:01:53.274676 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:53.274648    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"
Nov 02 16:01:53.274899 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:53.274645    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt" event={"ID":"101cfa8d-0553-4180-94e7-c9ba2ca41497","Type":"ContainerDied","Data":"ace1669a1e852256931b86e058378e02045526bcc2e6f26f041c802664b24a51"}
Nov 02 16:01:53.274899 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:53.274785    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace1669a1e852256931b86e058378e02045526bcc2e6f26f041c802664b24a51"
Nov 02 16:01:59.664010 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.663971    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"]
Nov 02 16:01:59.665156 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.665126    5041 cpu_manager.go:414] "containerMap: removing container" podUID="101cfa8d-0553-4180-94e7-c9ba2ca41497" containerName="revoke-keys"
Nov 02 16:01:59.665156 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.665158    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101cfa8d-0553-4180-94e7-c9ba2ca41497" containerName="revoke-keys"
Nov 02 16:01:59.665359 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.665346    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="101cfa8d-0553-4180-94e7-c9ba2ca41497" containerName="revoke-keys"
Nov 02 16:01:59.674532 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.674495    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.689752 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.689724    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"]
Nov 02 16:01:59.697572 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.697511    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:01:59.697572 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.697531    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:01:59.697572 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.697564    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:01:59.697826 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.697581    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:01:59.744035 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.744003    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.744219 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.744050    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwztx\" (UniqueName: \"kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.744219 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.744203    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.845546 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.845515    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.845733 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.845557    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwztx\" (UniqueName: \"kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.845733 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.845603    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.846237 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.846216    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.847924 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.847898    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.868720 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.868692    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwztx\" (UniqueName: \"kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx\") pod \"maas-api-revoke-sub-5a29efb3-6dcfx\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:01:59.989496 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:01:59.989471    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:02:00.287904 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:00.287871    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"]
Nov 02 16:02:00.288289 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:02:00.288266    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0759f3a_1a6b_4c32_bb13_5b234b007117.slice/crio-7a427af7c1497d9cd427c6519ce424c98afdf4fddf69bb36bcd59cdb9a7cdcd7 WatchSource:0}: Error finding container 7a427af7c1497d9cd427c6519ce424c98afdf4fddf69bb36bcd59cdb9a7cdcd7: Status 404 returned error can't find the container with id 7a427af7c1497d9cd427c6519ce424c98afdf4fddf69bb36bcd59cdb9a7cdcd7
Nov 02 16:02:00.321414 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:00.321385    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx" event={"ID":"d0759f3a-1a6b-4c32-bb13-5b234b007117","Type":"ContainerStarted","Data":"7a427af7c1497d9cd427c6519ce424c98afdf4fddf69bb36bcd59cdb9a7cdcd7"}
Nov 02 16:02:01.332767 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:01.332716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx" event={"ID":"d0759f3a-1a6b-4c32-bb13-5b234b007117","Type":"ContainerStarted","Data":"13fffb0d65ee6343d8ca029127d86c6fa6738a69952ee665e0071ca45ca744a0"}
Nov 02 16:02:01.379927 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:01.379856    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx" podStartSLOduration=2.379838205 podStartE2EDuration="2.379838205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:01.373612747 +0000 UTC m=+3973.631682212" watchObservedRunningTime="2026-11-02 16:02:01.379838205 +0000 UTC m=+3973.637907659"
Nov 02 16:02:02.388390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:02.387427    5041 generic.go:356] "Generic (PLEG): container finished" podID="d0759f3a-1a6b-4c32-bb13-5b234b007117" containerID="13fffb0d65ee6343d8ca029127d86c6fa6738a69952ee665e0071ca45ca744a0" exitCode=0
Nov 02 16:02:02.388390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:02.387589    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx" event={"ID":"d0759f3a-1a6b-4c32-bb13-5b234b007117","Type":"ContainerDied","Data":"13fffb0d65ee6343d8ca029127d86c6fa6738a69952ee665e0071ca45ca744a0"}
Nov 02 16:02:03.051829 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.051791    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"]
Nov 02 16:02:03.065047 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.065016    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h42mx"]
Nov 02 16:02:03.737001 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.736975    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:02:03.850792 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.850763    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca\") pod \"d0759f3a-1a6b-4c32-bb13-5b234b007117\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") "
Nov 02 16:02:03.850967 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.850900    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca\") pod \"d0759f3a-1a6b-4c32-bb13-5b234b007117\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") "
Nov 02 16:02:03.850967 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.850919    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx\" (UniqueName: \"kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx\") pod \"d0759f3a-1a6b-4c32-bb13-5b234b007117\" (UID: \"d0759f3a-1a6b-4c32-bb13-5b234b007117\") "
Nov 02 16:02:03.851265 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.851241    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca" pod "d0759f3a-1a6b-4c32-bb13-5b234b007117" (UID: "d0759f3a-1a6b-4c32-bb13-5b234b007117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:02:03.852985 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.852953    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca" pod "d0759f3a-1a6b-4c32-bb13-5b234b007117" (UID: "d0759f3a-1a6b-4c32-bb13-5b234b007117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:02:03.853115 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.853060    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx" pod "d0759f3a-1a6b-4c32-bb13-5b234b007117" (UID: "d0759f3a-1a6b-4c32-bb13-5b234b007117"). InnerVolumeSpecName "kube-api-access-fwztx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:02:03.886715 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.886690    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6311d1-f26b-41b2-b68a-23a8f309d712" path="/var/lib/kubelet/pods/6e6311d1-f26b-41b2-b68a-23a8f309d712/volumes"
Nov 02 16:02:03.951607 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.951573    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:02:03.951607 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.951603    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0759f3a-1a6b-4c32-bb13-5b234b007117-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:02:03.951772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:03.951619    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwztx\" (UniqueName: \"kubernetes.io/projected/d0759f3a-1a6b-4c32-bb13-5b234b007117-kube-api-access-fwztx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:02:04.413176 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:04.413147    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"
Nov 02 16:02:04.413346 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:04.413169    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx" event={"ID":"d0759f3a-1a6b-4c32-bb13-5b234b007117","Type":"ContainerDied","Data":"7a427af7c1497d9cd427c6519ce424c98afdf4fddf69bb36bcd59cdb9a7cdcd7"}
Nov 02 16:02:04.413346 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:04.413213    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a427af7c1497d9cd427c6519ce424c98afdf4fddf69bb36bcd59cdb9a7cdcd7"
Nov 02 16:02:06.767810 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:02:06.767758    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:02:18.681380 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.681332    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"]
Nov 02 16:02:18.692616 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.692586    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xcjc"]
Nov 02 16:02:18.729964 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.729928    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"]
Nov 02 16:02:18.730449 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.730428    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d0759f3a-1a6b-4c32-bb13-5b234b007117" containerName="revoke-keys"
Nov 02 16:02:18.730449 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.730444    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0759f3a-1a6b-4c32-bb13-5b234b007117" containerName="revoke-keys"
Nov 02 16:02:18.730673 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.730520    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0759f3a-1a6b-4c32-bb13-5b234b007117" containerName="revoke-keys"
Nov 02 16:02:18.734969 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.734947    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.749458 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.749436    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:02:18.749570 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.749521    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:02:18.752008 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.751988    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:02:18.755900 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.755872    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"]
Nov 02 16:02:18.850542 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.850511    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:02:18.887025 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.886995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgw5m\" (UniqueName: \"kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.887181 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.887112    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.887181 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.887143    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.987945 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.987882    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgw5m\" (UniqueName: \"kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.988058 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.987967    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.988058 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.987995    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.988663 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.988643    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:18.990294 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:18.990274    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:19.011329 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:19.011300    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgw5m\" (UniqueName: \"kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m\") pod \"maas-api-revoke-sub-3d4156f1-8sq8j\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:19.051108 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:19.051085    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:19.354242 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:19.354214    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"]
Nov 02 16:02:19.355224 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:02:19.355196    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3dc468_fd23_46e7_ae5b_52c100982e0c.slice/crio-86c6c55d7561702bae03c97c6b24d2a58572bedc269e21f615c2365f8e372b05 WatchSource:0}: Error finding container 86c6c55d7561702bae03c97c6b24d2a58572bedc269e21f615c2365f8e372b05: Status 404 returned error can't find the container with id 86c6c55d7561702bae03c97c6b24d2a58572bedc269e21f615c2365f8e372b05
Nov 02 16:02:19.500744 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:19.500716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j" event={"ID":"af3dc468-fd23-46e7-ae5b-52c100982e0c","Type":"ContainerStarted","Data":"86c6c55d7561702bae03c97c6b24d2a58572bedc269e21f615c2365f8e372b05"}
Nov 02 16:02:19.886565 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:19.886483    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2d5094-c50c-453a-af22-68b79f1588dd" path="/var/lib/kubelet/pods/bd2d5094-c50c-453a-af22-68b79f1588dd/volumes"
Nov 02 16:02:20.507925 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:20.507892    5041 generic.go:356] "Generic (PLEG): container finished" podID="af3dc468-fd23-46e7-ae5b-52c100982e0c" containerID="f5c917c4bc5719daa053ea5299ef7541e712a248f03171558ff40ee5fcc9b401" exitCode=0
Nov 02 16:02:20.508102 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:20.507976    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j" event={"ID":"af3dc468-fd23-46e7-ae5b-52c100982e0c","Type":"ContainerDied","Data":"f5c917c4bc5719daa053ea5299ef7541e712a248f03171558ff40ee5fcc9b401"}
Nov 02 16:02:21.661698 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.661674    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:21.813992 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.813908    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca\") pod \"af3dc468-fd23-46e7-ae5b-52c100982e0c\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") "
Nov 02 16:02:21.813992 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.813963    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca\") pod \"af3dc468-fd23-46e7-ae5b-52c100982e0c\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") "
Nov 02 16:02:21.814190 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.814015    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m\" (UniqueName: \"kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m\") pod \"af3dc468-fd23-46e7-ae5b-52c100982e0c\" (UID: \"af3dc468-fd23-46e7-ae5b-52c100982e0c\") "
Nov 02 16:02:21.814374 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.814331    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca" pod "af3dc468-fd23-46e7-ae5b-52c100982e0c" (UID: "af3dc468-fd23-46e7-ae5b-52c100982e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:02:21.816157 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.816132    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca" pod "af3dc468-fd23-46e7-ae5b-52c100982e0c" (UID: "af3dc468-fd23-46e7-ae5b-52c100982e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:02:21.816248 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.816134    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m" pod "af3dc468-fd23-46e7-ae5b-52c100982e0c" (UID: "af3dc468-fd23-46e7-ae5b-52c100982e0c"). InnerVolumeSpecName "kube-api-access-rgw5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:02:21.915205 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.915181    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:02:21.915205 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.915205    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3dc468-fd23-46e7-ae5b-52c100982e0c-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:02:21.915344 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:21.915215    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgw5m\" (UniqueName: \"kubernetes.io/projected/af3dc468-fd23-46e7-ae5b-52c100982e0c-kube-api-access-rgw5m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:02:22.521259 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:22.521220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j" event={"ID":"af3dc468-fd23-46e7-ae5b-52c100982e0c","Type":"ContainerDied","Data":"86c6c55d7561702bae03c97c6b24d2a58572bedc269e21f615c2365f8e372b05"}
Nov 02 16:02:22.521259 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:22.521249    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"
Nov 02 16:02:22.521506 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:22.521272    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c6c55d7561702bae03c97c6b24d2a58572bedc269e21f615c2365f8e372b05"
Nov 02 16:02:34.294156 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.294070    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"]
Nov 02 16:02:34.294629 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.294580    5041 cpu_manager.go:414] "containerMap: removing container" podUID="af3dc468-fd23-46e7-ae5b-52c100982e0c" containerName="revoke-keys"
Nov 02 16:02:34.294629 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.294596    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3dc468-fd23-46e7-ae5b-52c100982e0c" containerName="revoke-keys"
Nov 02 16:02:34.294770 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.294678    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3dc468-fd23-46e7-ae5b-52c100982e0c" containerName="revoke-keys"
Nov 02 16:02:34.305395 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.297633    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.312583 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.312545    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fee58f-openshift-default-dockercfg-hdgj4\""
Nov 02 16:02:34.328348 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7l5\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328515 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328386    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328515 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328415    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328515 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328480    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328515 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328503    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328789 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328538    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328789 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328563    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328789 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328631    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.328789 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.328690    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.329982 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.329950    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"]
Nov 02 16:02:34.429440 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429397    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429656 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429457    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429656 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429493    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429656 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429656 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429579    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429656 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429631    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429925 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429685    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7l5\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429925 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429726    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.429925 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429860    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.430088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429923    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.430088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.429929    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.430261 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.430233    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.430431 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.430238    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.430431 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.430331    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.431975 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.431950    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.432278 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.432257    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.454045 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.453999    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7l5\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.454463 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.454439    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token\") pod \"e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") " pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.623872 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.623786    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:34.940118 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.940085    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"]
Nov 02 16:02:34.941314 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:02:34.941288    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd799063_5c86_4304_8d13_ec49d350decf.slice/crio-0b7809523e2252d79cb76d4b6b79d6e6777132629e38fe32295d1d028757f492 WatchSource:0}: Error finding container 0b7809523e2252d79cb76d4b6b79d6e6777132629e38fe32295d1d028757f492: Status 404 returned error can't find the container with id 0b7809523e2252d79cb76d4b6b79d6e6777132629e38fe32295d1d028757f492
Nov 02 16:02:34.943670 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.943638    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:02:34.943762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.943701    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:02:34.943762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:34.943738    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:02:35.601430 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:35.601395    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" event={"ID":"bd799063-5c86-4304-8d13-ec49d350decf","Type":"ContainerStarted","Data":"859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"}
Nov 02 16:02:35.601851 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:35.601449    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" event={"ID":"bd799063-5c86-4304-8d13-ec49d350decf","Type":"ContainerStarted","Data":"0b7809523e2252d79cb76d4b6b79d6e6777132629e38fe32295d1d028757f492"}
Nov 02 16:02:35.624683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:35.624654    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:35.630051 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:35.630031    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:35.657549 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:35.657507    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podStartSLOduration=1.657495484 podStartE2EDuration="1.657495484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:35.655943369 +0000 UTC m=+4007.914012831" watchObservedRunningTime="2026-11-02 16:02:35.657495484 +0000 UTC m=+4007.915564933"
Nov 02 16:02:36.607358 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:36.607321    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:36.608200 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:36.608183    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:36.769248 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:02:36.769212    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:02:39.547771 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.547733    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"]
Nov 02 16:02:39.551712 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.551682    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.567991 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.567967    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fee58f-serving-cert\""
Nov 02 16:02:39.578236 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.578201    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 16:02:39.578236 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.578209    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:02:39.578236 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.578230    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:02:39.578460 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.578284    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cz4f2\""
Nov 02 16:02:39.579848 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.579829    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.579938 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.579869    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnms\" (UniqueName: \"kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.579938 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.579900    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.580028 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.580015    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.595273 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.595231    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"]
Nov 02 16:02:39.680762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.680712    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnms\" (UniqueName: \"kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.680971 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.680795    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.680971 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.680826    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.681101 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.680970    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.681164 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:02:39.681098    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-serving-cert: secret "maas-api-e2e-worker-single-main-fee58f-serving-cert" not found
Nov 02 16:02:39.681231 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:02:39.681173    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls podName:481104b4-92da-4765-9415-d3504f7361fb nodeName:}" failed. No retries permitted until 2026-11-02 16:02:40.18115228 +0000 UTC m=+4012.439221713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls") pod "maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" (UID: "481104b4-92da-4765-9415-d3504f7361fb") : secret "maas-api-e2e-worker-single-main-fee58f-serving-cert" not found
Nov 02 16:02:39.681501 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.681479    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.683464 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.683444    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:39.742874 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:39.742839    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnms\" (UniqueName: \"kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:40.114881 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.114842    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"]
Nov 02 16:02:40.118838 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.118815    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.138419 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.138397    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-761e43f8-dockercfg-qb2v7\""
Nov 02 16:02:40.141927 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.141908    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fee58f\""
Nov 02 16:02:40.146271 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.146244    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"]
Nov 02 16:02:40.193680 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.193639    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:40.194062 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.194030    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.194228 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.194209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x8f\" (UniqueName: \"kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f\") pod \"payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.212142 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.210955    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:40.216753 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.216722    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"]
Nov 02 16:02:40.221394 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.221234    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.242140 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.242109    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"]
Nov 02 16:02:40.295577 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.295540    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbd4r\" (UniqueName: \"kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r\") pod \"payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.295751 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.295587    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.295751 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.295689    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x8f\" (UniqueName: \"kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f\") pod \"payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.295832 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.295765    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.296333 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.296314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.321109 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.321074    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x8f\" (UniqueName: \"kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f\") pod \"payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.396733 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.396653    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbd4r\" (UniqueName: \"kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r\") pod \"payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.396733 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.396717    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.397357 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.397338    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.419481 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.419455    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbd4r\" (UniqueName: \"kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r\") pod \"payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.432565 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.432543    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:40.466430 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.466395    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:40.537980 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.537949    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:40.736623 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.736594    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"]
Nov 02 16:02:40.737571 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:02:40.737543    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee348c0_6881_454d_b0c2_e5655e720c32.slice/crio-8ff765b4f43e644ae0cf5b38545344bb1ed4eb4d2b94fc642e5c439e09a13b2f WatchSource:0}: Error finding container 8ff765b4f43e644ae0cf5b38545344bb1ed4eb4d2b94fc642e5c439e09a13b2f: Status 404 returned error can't find the container with id 8ff765b4f43e644ae0cf5b38545344bb1ed4eb4d2b94fc642e5c439e09a13b2f
Nov 02 16:02:40.780344 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.780319    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"]
Nov 02 16:02:40.783625 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:02:40.783598    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481104b4_92da_4765_9415_d3504f7361fb.slice/crio-809eeb56e88a13a7960c9ea364fe6c6e400b616ac808974b50d9b050ec41f328 WatchSource:0}: Error finding container 809eeb56e88a13a7960c9ea364fe6c6e400b616ac808974b50d9b050ec41f328: Status 404 returned error can't find the container with id 809eeb56e88a13a7960c9ea364fe6c6e400b616ac808974b50d9b050ec41f328
Nov 02 16:02:40.785491 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.785474    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 16:02:40.875390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:40.875272    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"]
Nov 02 16:02:41.649714 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.649656    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" event={"ID":"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c","Type":"ContainerStarted","Data":"df1e795e9355debefa6bb07b482d2445445acd0ebfd2c45e20f8d7d7336ec763"}
Nov 02 16:02:41.649959 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.649729    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" event={"ID":"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c","Type":"ContainerStarted","Data":"7036a62f0db8071928a33e332c9f6d6f61586f6712310799054169886c82f257"}
Nov 02 16:02:41.650882 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.650857    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:41.652882 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.652822    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" event={"ID":"481104b4-92da-4765-9415-d3504f7361fb","Type":"ContainerStarted","Data":"5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c"}
Nov 02 16:02:41.653018 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.652888    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:41.653018 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.652915    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" event={"ID":"481104b4-92da-4765-9415-d3504f7361fb","Type":"ContainerStarted","Data":"809eeb56e88a13a7960c9ea364fe6c6e400b616ac808974b50d9b050ec41f328"}
Nov 02 16:02:41.654601 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.654578    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" event={"ID":"6ee348c0-6881-454d-b0c2-e5655e720c32","Type":"ContainerStarted","Data":"a91d81ee1ff2264b325289e35234fa066076b4c9c1775fc6c56815c55b114081"}
Nov 02 16:02:41.654674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.654616    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" event={"ID":"6ee348c0-6881-454d-b0c2-e5655e720c32","Type":"ContainerStarted","Data":"8ff765b4f43e644ae0cf5b38545344bb1ed4eb4d2b94fc642e5c439e09a13b2f"}
Nov 02 16:02:41.654674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.654657    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:41.690323 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.690270    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" podStartSLOduration=1.690254877 podStartE2EDuration="1.690254877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:41.689297494 +0000 UTC m=+4013.947366943" watchObservedRunningTime="2026-11-02 16:02:41.690254877 +0000 UTC m=+4013.948324328"
Nov 02 16:02:41.732629 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.732565    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" podStartSLOduration=2.411333144 podStartE2EDuration="2.732549679s" totalImagesPullingTime="321.216535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:41.729516686 +0000 UTC m=+4013.987586137" watchObservedRunningTime="2026-11-02 16:02:41.732549679 +0000 UTC m=+4013.990619129"
Nov 02 16:02:41.769553 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:41.769502    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" podStartSLOduration=1.769490024 podStartE2EDuration="1.769490024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:41.767855288 +0000 UTC m=+4014.025924737" watchObservedRunningTime="2026-11-02 16:02:41.769490024 +0000 UTC m=+4014.027559474"
Nov 02 16:02:47.666962 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:47.666926    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:02:52.649584 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:52.649534    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:02:52.650043 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:52.649605    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:02:52.664677 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:52.664640    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:02:53.672595 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.672557    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:02:53.736986 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.736952    5041 generic.go:356] "Generic (PLEG): container finished" podID="bd799063-5c86-4304-8d13-ec49d350decf" containerID="859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427" exitCode=0
Nov 02 16:02:53.737170 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.737029    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" event={"ID":"bd799063-5c86-4304-8d13-ec49d350decf","Type":"ContainerDied","Data":"859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"}
Nov 02 16:02:53.737589 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.737565    5041 scope.go:122] "RemoveContainer" containerID="859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"
Nov 02 16:02:53.738984 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.738939    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:02:53.739109 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.739035    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:02:53.739109 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:53.739073    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:02:54.747343 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:54.747300    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" event={"ID":"bd799063-5c86-4304-8d13-ec49d350decf","Type":"ContainerStarted","Data":"2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05"}
Nov 02 16:02:55.631388 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.628739    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:02:55.631388 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.628886    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Nov 02 16:02:55.631388 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.628933    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Nov 02 16:02:55.644548 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.644510    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"]
Nov 02 16:02:55.647862 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.647836    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.673812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.673778    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fee58f\"/\"default-dockercfg-l67wv\""
Nov 02 16:02:55.674725 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.674477    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fee58f\"/\"kube-root-ca.crt\""
Nov 02 16:02:55.675083 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.675036    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fee58f\"/\"simulator-single-main-fee58f-kserve-self-signed-certs\""
Nov 02 16:02:55.675241 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.675110    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fee58f\"/\"openshift-service-ca.crt\""
Nov 02 16:02:55.679619 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.679594    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"]
Nov 02 16:02:55.778806 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.778768    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.779197 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.778816    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.779197 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.778888    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.779197 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.778954    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.779197 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.779005    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jwx\" (UniqueName: \"kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.779197 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.779029    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.880548 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.880505    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.880765 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.880583    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.880765 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.880605    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.880765 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.880624    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.880765 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.880674    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jwx\" (UniqueName: \"kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.880765 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.880704    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.881116 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.881091    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.881264 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.881244    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.881347 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.881323    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.883077 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.883007    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.883660 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.883641    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.903256 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.903233    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jwx\" (UniqueName: \"kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx\") pod \"simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") " pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:55.962609 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:55.962574    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:56.283850 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.283812    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"]
Nov 02 16:02:56.285288 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:02:56.285262    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb921ea2_bcfd_45b4_b81b_e8dc26a2c590.slice/crio-8ff427a1d3e2ba3d2cb0f67a7fe3f5668e6640984144473cafc632228a1f23e8 WatchSource:0}: Error finding container 8ff427a1d3e2ba3d2cb0f67a7fe3f5668e6640984144473cafc632228a1f23e8: Status 404 returned error can't find the container with id 8ff427a1d3e2ba3d2cb0f67a7fe3f5668e6640984144473cafc632228a1f23e8
Nov 02 16:02:56.637881 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.637783    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Nov 02 16:02:56.637881 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.637856    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Nov 02 16:02:56.761814 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.761769    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5" event={"ID":"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590","Type":"ContainerStarted","Data":"deb07e88fd93693fc82a48999c024c5471e08aba319a0a522b5376e26f7de138"}
Nov 02 16:02:56.762008 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.761824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5" event={"ID":"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590","Type":"ContainerStarted","Data":"8ff427a1d3e2ba3d2cb0f67a7fe3f5668e6640984144473cafc632228a1f23e8"}
Nov 02 16:02:56.803376 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.803316    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5" podStartSLOduration=1.803302839 podStartE2EDuration="1.803302839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:56.801709826 +0000 UTC m=+4029.059779276" watchObservedRunningTime="2026-11-02 16:02:56.803302839 +0000 UTC m=+4029.061372342"
Nov 02 16:02:56.963628 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:56.963588    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:02:57.631529 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:57.631493    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Nov 02 16:02:57.631860 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:57.631828    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Nov 02 16:02:58.637390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:58.635765    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Nov 02 16:02:58.637390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:58.635834    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Nov 02 16:02:59.624621 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:59.624577    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body=
Nov 02 16:02:59.624817 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:02:59.624640    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused"
Nov 02 16:03:00.627838 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:00.627804    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:03:00.628232 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:00.628117    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:03:00.628894 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:00.628870    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:03:02.037865 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:02.037824    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"]
Nov 02 16:03:02.048798 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:02.048761    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-2xk9t"]
Nov 02 16:03:03.886747 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:03.886710    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f9fed8-42ab-4f2a-9069-61cd221b36ef" path="/var/lib/kubelet/pods/c2f9fed8-42ab-4f2a-9069-61cd221b36ef/volumes"
Nov 02 16:03:05.963687 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:05.963655    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:03:05.976466 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:05.976438    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:03:06.770066 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:03:06.770022    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:03:06.841542 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:06.841505    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:03:22.029188 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:22.029153    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"]
Nov 02 16:03:22.040153 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:22.040125    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-c2bpx"]
Nov 02 16:03:23.886608 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:23.886572    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53c835e-2d87-491f-9e52-6b054ca44032" path="/var/lib/kubelet/pods/c53c835e-2d87-491f-9e52-6b054ca44032/volumes"
Nov 02 16:03:34.032079 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:34.032037    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"]
Nov 02 16:03:34.042250 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:34.042217    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggdhn"]
Nov 02 16:03:35.886889 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:35.886841    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adba059-fb39-4fee-9e91-368c18837d73" path="/var/lib/kubelet/pods/4adba059-fb39-4fee-9e91-368c18837d73/volumes"
Nov 02 16:03:36.768535 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:03:36.768497    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:03:52.690457 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:03:52.690421    5041 scope.go:122] "RemoveContainer" containerID="e0c0e9abc986517d749e17127101907905e4e028a10668e81dab613a77abaaec"
Nov 02 16:04:03.031267 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:03.031175    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"]
Nov 02 16:04:03.042159 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:03.042124    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zx44h"]
Nov 02 16:04:03.154978 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:03.154936    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"]
Nov 02 16:04:03.886804 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:03.886772    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c204c6af-fb7d-4198-8959-8cf16cff1883" path="/var/lib/kubelet/pods/c204c6af-fb7d-4198-8959-8cf16cff1883/volumes"
Nov 02 16:04:06.769416 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:04:06.769351    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:04:08.924422 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:08.924381    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"]
Nov 02 16:04:08.958936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:08.958901    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"]
Nov 02 16:04:11.601243 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:11.601199    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"]
Nov 02 16:04:11.601785 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:11.601476    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" podUID="481104b4-92da-4765-9415-d3504f7361fb" containerName="maas-api" containerID="cri-o://5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c" gracePeriod=30
Nov 02 16:04:12.088033 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.088006    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:04:12.176938 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.176908    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca\") pod \"481104b4-92da-4765-9415-d3504f7361fb\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") "
Nov 02 16:04:12.177137 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.176997    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\") pod \"481104b4-92da-4765-9415-d3504f7361fb\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") "
Nov 02 16:04:12.177137 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.177055    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs\") pod \"481104b4-92da-4765-9415-d3504f7361fb\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") "
Nov 02 16:04:12.177137 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.177076    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms\" (UniqueName: \"kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms\") pod \"481104b4-92da-4765-9415-d3504f7361fb\" (UID: \"481104b4-92da-4765-9415-d3504f7361fb\") "
Nov 02 16:04:12.177342 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.177294    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca" pod "481104b4-92da-4765-9415-d3504f7361fb" (UID: "481104b4-92da-4765-9415-d3504f7361fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:12.179339 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.179308    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls" pod "481104b4-92da-4765-9415-d3504f7361fb" (UID: "481104b4-92da-4765-9415-d3504f7361fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:12.179465 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.179336    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs" pod "481104b4-92da-4765-9415-d3504f7361fb" (UID: "481104b4-92da-4765-9415-d3504f7361fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:12.179465 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.179316    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms" pod "481104b4-92da-4765-9415-d3504f7361fb" (UID: "481104b4-92da-4765-9415-d3504f7361fb"). InnerVolumeSpecName "kube-api-access-rdnms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:12.246666 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.246630    5041 generic.go:356] "Generic (PLEG): container finished" podID="481104b4-92da-4765-9415-d3504f7361fb" containerID="5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c" exitCode=0
Nov 02 16:04:12.246905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.246681    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"
Nov 02 16:04:12.246905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.246729    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" event={"ID":"481104b4-92da-4765-9415-d3504f7361fb","Type":"ContainerDied","Data":"5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c"}
Nov 02 16:04:12.246905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.246793    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c" event={"ID":"481104b4-92da-4765-9415-d3504f7361fb","Type":"ContainerDied","Data":"809eeb56e88a13a7960c9ea364fe6c6e400b616ac808974b50d9b050ec41f328"}
Nov 02 16:04:12.246905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.246823    5041 scope.go:122] "RemoveContainer" containerID="5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c"
Nov 02 16:04:12.257486 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.257434    5041 scope.go:122] "RemoveContainer" containerID="5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c"
Nov 02 16:04:12.257705 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:04:12.257685    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c\": container with ID starting with 5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c not found: ID does not exist" containerID="5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c"
Nov 02 16:04:12.257767 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.257714    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c"} err="failed to get container status \"5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c\": rpc error: code = NotFound desc = could not find container \"5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c\": container with ID starting with 5f7822e618e009fe72c7ae386eb63ddac4b04738f15d7b3fbe909a15c340099c not found: ID does not exist"
Nov 02 16:04:12.278547 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.278526    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:12.278547 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.278547    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/481104b4-92da-4765-9415-d3504f7361fb-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:12.278689 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.278558    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdnms\" (UniqueName: \"kubernetes.io/projected/481104b4-92da-4765-9415-d3504f7361fb-kube-api-access-rdnms\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:12.278689 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.278567    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481104b4-92da-4765-9415-d3504f7361fb-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:12.298761 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.298735    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"]
Nov 02 16:04:12.301729 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:12.301701    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fee58f-7d95cff487-sk24c"]
Nov 02 16:04:13.029415 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:13.029377    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"]
Nov 02 16:04:13.040219 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:13.040189    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b9zzz"]
Nov 02 16:04:13.887308 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:13.887269    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b" path="/var/lib/kubelet/pods/362e176b-9ef8-4dfd-b1f8-16f7e68c8c3b/volumes"
Nov 02 16:04:13.887788 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:13.887771    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481104b4-92da-4765-9415-d3504f7361fb" path="/var/lib/kubelet/pods/481104b4-92da-4765-9415-d3504f7361fb/volumes"
Nov 02 16:04:13.925165 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:13.925133    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerName="payload-processing" containerID="cri-o://df1e795e9355debefa6bb07b482d2445445acd0ebfd2c45e20f8d7d7336ec763" gracePeriod=25
Nov 02 16:04:13.959439 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:13.959404    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerName="payload-pre-processing" containerID="cri-o://a91d81ee1ff2264b325289e35234fa066076b4c9c1775fc6c56815c55b114081" gracePeriod=25
Nov 02 16:04:18.176638 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.176572    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5" podUID="cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" containerName="main" containerID="cri-o://deb07e88fd93693fc82a48999c024c5471e08aba319a0a522b5376e26f7de138" gracePeriod=45
Nov 02 16:04:18.293410 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.293376    5041 generic.go:356] "Generic (PLEG): container finished" podID="cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" containerID="deb07e88fd93693fc82a48999c024c5471e08aba319a0a522b5376e26f7de138" exitCode=0
Nov 02 16:04:18.293539 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.293469    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5" event={"ID":"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590","Type":"ContainerDied","Data":"deb07e88fd93693fc82a48999c024c5471e08aba319a0a522b5376e26f7de138"}
Nov 02 16:04:18.431731 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.431666    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:04:18.540750 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.540720    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache\") pod \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") "
Nov 02 16:04:18.540961 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.540788    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home\") pod \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") "
Nov 02 16:04:18.540961 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.540819    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm\") pod \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") "
Nov 02 16:04:18.540961 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.540838    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx\" (UniqueName: \"kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx\") pod \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") "
Nov 02 16:04:18.540961 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.540865    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir\") pod \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") "
Nov 02 16:04:18.540961 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.540937    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs\") pod \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\" (UID: \"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590\") "
Nov 02 16:04:18.541224 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.541046    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache" pod "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" (UID: "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:18.541224 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.541100    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home" pod "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" (UID: "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:18.541224 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.541157    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir" pod "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" (UID: "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:18.541674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.541651    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:18.541674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.541673    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:18.541855 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.541683    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:18.543255 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.543231    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs" pod "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" (UID: "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:18.543391 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.543274    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx" pod "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" (UID: "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590"). InnerVolumeSpecName "kube-api-access-g5jwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:18.543600 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.543583    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm" pod "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" (UID: "cb921ea2-bcfd-45b4-b81b-e8dc26a2c590"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:18.642707 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.642681    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:18.642707 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.642708    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jwx\" (UniqueName: \"kubernetes.io/projected/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-kube-api-access-g5jwx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:18.642882 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:18.642719    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:19.301088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.301055    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"
Nov 02 16:04:19.301621 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.301094    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5" event={"ID":"cb921ea2-bcfd-45b4-b81b-e8dc26a2c590","Type":"ContainerDied","Data":"8ff427a1d3e2ba3d2cb0f67a7fe3f5668e6640984144473cafc632228a1f23e8"}
Nov 02 16:04:19.301621 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.301145    5041 scope.go:122] "RemoveContainer" containerID="deb07e88fd93693fc82a48999c024c5471e08aba319a0a522b5376e26f7de138"
Nov 02 16:04:19.355271 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.353436    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"]
Nov 02 16:04:19.355271 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.353678    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy" containerID="cri-o://2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05" gracePeriod=30
Nov 02 16:04:19.377473 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.377427    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"]
Nov 02 16:04:19.383212 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.383128    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fee58f/simulator-single-main-fee58f-kserve-787f7cbf9-vj6q5"]
Nov 02 16:04:19.886668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:19.886634    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" path="/var/lib/kubelet/pods/cb921ea2-bcfd-45b4-b81b-e8dc26a2c590/volumes"
Nov 02 16:04:20.639536 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.639505    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"]
Nov 02 16:04:20.650762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.650732    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq8j"]
Nov 02 16:04:20.690783 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.690753    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"]
Nov 02 16:04:20.691199 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.691186    5041 cpu_manager.go:414] "containerMap: removing container" podUID="481104b4-92da-4765-9415-d3504f7361fb" containerName="maas-api"
Nov 02 16:04:20.691247 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.691201    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481104b4-92da-4765-9415-d3504f7361fb" containerName="maas-api"
Nov 02 16:04:20.691247 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.691209    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" containerName="main"
Nov 02 16:04:20.691247 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.691214    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" containerName="main"
Nov 02 16:04:20.691359 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.691308    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="481104b4-92da-4765-9415-d3504f7361fb" containerName="maas-api"
Nov 02 16:04:20.691359 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.691316    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb921ea2-bcfd-45b4-b81b-e8dc26a2c590" containerName="main"
Nov 02 16:04:20.694278 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.694261    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.708264 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.708231    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:04:20.708486 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.708462    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:04:20.708605 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.708494    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:04:20.708605 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.708538    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:04:20.722378 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.722342    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"]
Nov 02 16:04:20.867703 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.867674    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.867873 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.867739    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zb7j\" (UniqueName: \"kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.867873 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.867848    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.969306 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.969277    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.969487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.969384    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zb7j\" (UniqueName: \"kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.969487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.969433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.970054 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.970030    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.971835 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.971809    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:20.992047 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:20.992019    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zb7j\" (UniqueName: \"kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j\") pod \"maas-api-revoke-sub-3d4156f1-6vhl4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:21.006740 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:21.006720    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:21.303644 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:21.303612    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"]
Nov 02 16:04:21.304869 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:04:21.304842    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30eed75c_39fc_4e1d_96ea_8bc58db718d4.slice/crio-30940d7e85caebaf3ace14f715ce23da489708d127ae31efe5cb10901b84c8ac WatchSource:0}: Error finding container 30940d7e85caebaf3ace14f715ce23da489708d127ae31efe5cb10901b84c8ac: Status 404 returned error can't find the container with id 30940d7e85caebaf3ace14f715ce23da489708d127ae31efe5cb10901b84c8ac
Nov 02 16:04:21.315492 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:21.315466    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4" event={"ID":"30eed75c-39fc-4e1d-96ea-8bc58db718d4","Type":"ContainerStarted","Data":"30940d7e85caebaf3ace14f715ce23da489708d127ae31efe5cb10901b84c8ac"}
Nov 02 16:04:21.886928 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:21.886864    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3dc468-fd23-46e7-ae5b-52c100982e0c" path="/var/lib/kubelet/pods/af3dc468-fd23-46e7-ae5b-52c100982e0c/volumes"
Nov 02 16:04:22.323485 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:22.323452    5041 generic.go:356] "Generic (PLEG): container finished" podID="30eed75c-39fc-4e1d-96ea-8bc58db718d4" containerID="a65719a1d420c062997bf8030d47c6deb68fd50cd665ab5fd6cb6998956411a0" exitCode=0
Nov 02 16:04:22.323621 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:22.323543    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4" event={"ID":"30eed75c-39fc-4e1d-96ea-8bc58db718d4","Type":"ContainerDied","Data":"a65719a1d420c062997bf8030d47c6deb68fd50cd665ab5fd6cb6998956411a0"}
Nov 02 16:04:22.663225 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:22.663147    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:04:23.461226 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.461204    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:23.596004 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.595942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j\" (UniqueName: \"kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j\") pod \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") "
Nov 02 16:04:23.596004 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.595990    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca\") pod \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") "
Nov 02 16:04:23.596156 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.596138    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca\") pod \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\" (UID: \"30eed75c-39fc-4e1d-96ea-8bc58db718d4\") "
Nov 02 16:04:23.596443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.596418    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca" pod "30eed75c-39fc-4e1d-96ea-8bc58db718d4" (UID: "30eed75c-39fc-4e1d-96ea-8bc58db718d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:23.596515 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.596489    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:23.598095 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.598072    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j" pod "30eed75c-39fc-4e1d-96ea-8bc58db718d4" (UID: "30eed75c-39fc-4e1d-96ea-8bc58db718d4"). InnerVolumeSpecName "kube-api-access-9zb7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:23.598195 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.598103    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca" pod "30eed75c-39fc-4e1d-96ea-8bc58db718d4" (UID: "30eed75c-39fc-4e1d-96ea-8bc58db718d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:23.671385 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.671339    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:04:23.697318 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.697297    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eed75c-39fc-4e1d-96ea-8bc58db718d4-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:23.697431 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:23.697321    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zb7j\" (UniqueName: \"kubernetes.io/projected/30eed75c-39fc-4e1d-96ea-8bc58db718d4-kube-api-access-9zb7j\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:24.338017 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:24.337992    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"
Nov 02 16:04:24.338184 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:24.338035    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4" event={"ID":"30eed75c-39fc-4e1d-96ea-8bc58db718d4","Type":"ContainerDied","Data":"30940d7e85caebaf3ace14f715ce23da489708d127ae31efe5cb10901b84c8ac"}
Nov 02 16:04:24.338184 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:24.338086    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30940d7e85caebaf3ace14f715ce23da489708d127ae31efe5cb10901b84c8ac"
Nov 02 16:04:25.028927 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.028901    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:04:25.211648 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211619    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.211831 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.211831 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211704    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.211831 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211738    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.211831 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211783    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211836    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211870    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211899    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211919    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211969    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.211999    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data\") pod \"bd799063-5c86-4304-8d13-ec49d350decf\" (UID: \"bd799063-5c86-4304-8d13-ec49d350decf\") "
Nov 02 16:04:25.212071 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212043    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:25.212455 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212140    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:25.212455 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212389    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:25.212542 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212500    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.212542 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212525    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.212627 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212539    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.212627 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.212554    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd799063-5c86-4304-8d13-ec49d350decf-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.213987 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.213967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 16:04:25.214115 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.214098    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:25.214420 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.214398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:25.214503 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.214403    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5" pod "bd799063-5c86-4304-8d13-ec49d350decf" (UID: "bd799063-5c86-4304-8d13-ec49d350decf"). InnerVolumeSpecName "kube-api-access-ql7l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:25.313682 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.313661    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd799063-5c86-4304-8d13-ec49d350decf-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.313682 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.313682    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7l5\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-kube-api-access-ql7l5\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.313812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.313692    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.313812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.313700    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd799063-5c86-4304-8d13-ec49d350decf-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.313812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.313709    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd799063-5c86-4304-8d13-ec49d350decf-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:25.347542 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.347518    5041 generic.go:356] "Generic (PLEG): container finished" podID="bd799063-5c86-4304-8d13-ec49d350decf" containerID="2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05" exitCode=0
Nov 02 16:04:25.347630 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.347565    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"
Nov 02 16:04:25.347630 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.347585    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" event={"ID":"bd799063-5c86-4304-8d13-ec49d350decf","Type":"ContainerDied","Data":"2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05"}
Nov 02 16:04:25.347717 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.347629    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb" event={"ID":"bd799063-5c86-4304-8d13-ec49d350decf","Type":"ContainerDied","Data":"0b7809523e2252d79cb76d4b6b79d6e6777132629e38fe32295d1d028757f492"}
Nov 02 16:04:25.347717 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.347652    5041 scope.go:122] "RemoveContainer" containerID="2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05"
Nov 02 16:04:25.358028 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.358006    5041 scope.go:122] "RemoveContainer" containerID="859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"
Nov 02 16:04:25.366827 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.366809    5041 scope.go:122] "RemoveContainer" containerID="2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05"
Nov 02 16:04:25.367098 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:04:25.367080    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05\": container with ID starting with 2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05 not found: ID does not exist" containerID="2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05"
Nov 02 16:04:25.367145 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.367106    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05"} err="failed to get container status \"2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05\": rpc error: code = NotFound desc = could not find container \"2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05\": container with ID starting with 2d39fd221be4d10cf2a478ae7594b5adc1ca27576d197f0a3813384db4620f05 not found: ID does not exist"
Nov 02 16:04:25.367145 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.367124    5041 scope.go:122] "RemoveContainer" containerID="859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"
Nov 02 16:04:25.367432 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:04:25.367379    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427\": container with ID starting with 859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427 not found: ID does not exist" containerID="859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"
Nov 02 16:04:25.367432 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.367409    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427"} err="failed to get container status \"859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427\": rpc error: code = NotFound desc = could not find container \"859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427\": container with ID starting with 859cc947acfe2f2d7c003e20c609a41765dfde407911352d169e8490c72e1427 not found: ID does not exist"
Nov 02 16:04:25.397626 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.397604    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"]
Nov 02 16:04:25.408430 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.408411    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fee58f-openshift-default-768fcf88c7fc6rb"]
Nov 02 16:04:25.886380 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:25.886342    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd799063-5c86-4304-8d13-ec49d350decf" path="/var/lib/kubelet/pods/bd799063-5c86-4304-8d13-ec49d350decf/volumes"
Nov 02 16:04:32.662336 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:32.662298    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:04:33.685508 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:33.685464    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:04:34.500053 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:34.498457    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:04:34.500053 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:34.498528    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:04:34.577316 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:34.577265    5041 generic.go:356] "Generic (PLEG): container finished" podID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerID="df1e795e9355debefa6bb07b482d2445445acd0ebfd2c45e20f8d7d7336ec763" exitCode=0
Nov 02 16:04:34.577761 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:34.577562    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" event={"ID":"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c","Type":"ContainerDied","Data":"df1e795e9355debefa6bb07b482d2445445acd0ebfd2c45e20f8d7d7336ec763"}
Nov 02 16:04:34.592153 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:34.592001    5041 generic.go:356] "Generic (PLEG): container finished" podID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerID="a91d81ee1ff2264b325289e35234fa066076b4c9c1775fc6c56815c55b114081" exitCode=0
Nov 02 16:04:34.592153 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:34.592096    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" event={"ID":"6ee348c0-6881-454d-b0c2-e5655e720c32","Type":"ContainerDied","Data":"a91d81ee1ff2264b325289e35234fa066076b4c9c1775fc6c56815c55b114081"}
Nov 02 16:04:35.220900 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.220538    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:04:35.261637 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.261591    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:04:35.353562 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.353460    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume\") pod \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") "
Nov 02 16:04:35.353747 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.353591    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r\" (UniqueName: \"kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r\") pod \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\" (UID: \"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c\") "
Nov 02 16:04:35.353747 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.353680    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume\") pod \"6ee348c0-6881-454d-b0c2-e5655e720c32\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") "
Nov 02 16:04:35.353896 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.353748    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f\" (UniqueName: \"kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f\") pod \"6ee348c0-6881-454d-b0c2-e5655e720c32\" (UID: \"6ee348c0-6881-454d-b0c2-e5655e720c32\") "
Nov 02 16:04:35.353896 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.353858    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume" pod "9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" (UID: "9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:35.354118 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.354094    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume" pod "6ee348c0-6881-454d-b0c2-e5655e720c32" (UID: "6ee348c0-6881-454d-b0c2-e5655e720c32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:35.354192 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.354097    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:35.356169 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.356144    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r" pod "9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" (UID: "9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c"). InnerVolumeSpecName "kube-api-access-fbd4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:35.356502 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.356480    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f" pod "6ee348c0-6881-454d-b0c2-e5655e720c32" (UID: "6ee348c0-6881-454d-b0c2-e5655e720c32"). InnerVolumeSpecName "kube-api-access-b8x8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:35.455042 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.454999    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbd4r\" (UniqueName: \"kubernetes.io/projected/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c-kube-api-access-fbd4r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:35.455042 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.455034    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee348c0-6881-454d-b0c2-e5655e720c32-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:35.455042 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.455044    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8x8f\" (UniqueName: \"kubernetes.io/projected/6ee348c0-6881-454d-b0c2-e5655e720c32-kube-api-access-b8x8f\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:04:35.607632 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.607545    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"
Nov 02 16:04:35.607632 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.607568    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v" event={"ID":"9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c","Type":"ContainerDied","Data":"7036a62f0db8071928a33e332c9f6d6f61586f6712310799054169886c82f257"}
Nov 02 16:04:35.607899 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.607670    5041 scope.go:122] "RemoveContainer" containerID="df1e795e9355debefa6bb07b482d2445445acd0ebfd2c45e20f8d7d7336ec763"
Nov 02 16:04:35.609175 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.609151    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"
Nov 02 16:04:35.609309 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.609158    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n" event={"ID":"6ee348c0-6881-454d-b0c2-e5655e720c32","Type":"ContainerDied","Data":"8ff765b4f43e644ae0cf5b38545344bb1ed4eb4d2b94fc642e5c439e09a13b2f"}
Nov 02 16:04:35.617795 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.617771    5041 scope.go:122] "RemoveContainer" containerID="a91d81ee1ff2264b325289e35234fa066076b4c9c1775fc6c56815c55b114081"
Nov 02 16:04:35.658120 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.658093    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"]
Nov 02 16:04:35.662100 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.662076    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fee58f-56ff6fgh7n"]
Nov 02 16:04:35.694108 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.694078    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"]
Nov 02 16:04:35.698827 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.698800    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fee58f-697595fcf2sj5v"]
Nov 02 16:04:35.885872 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.885797    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" path="/var/lib/kubelet/pods/6ee348c0-6881-454d-b0c2-e5655e720c32/volumes"
Nov 02 16:04:35.886176 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:35.886163    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" path="/var/lib/kubelet/pods/9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c/volumes"
Nov 02 16:04:36.768633 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:04:36.768594    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:04:39.042904 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:39.042863    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"]
Nov 02 16:04:39.053790 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:39.053755    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mzsfh"]
Nov 02 16:04:39.887080 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:39.887045    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4" path="/var/lib/kubelet/pods/a7943581-ce7c-4aaa-a314-c9a9e8e2a1a4/volumes"
Nov 02 16:04:52.720641 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:52.720601    5041 scope.go:122] "RemoveContainer" containerID="32aa15154db1cc2d9b131978e18c5c0b8b68939145841b9558201fa973d9cbb6"
Nov 02 16:04:52.730202 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:52.730179    5041 scope.go:122] "RemoveContainer" containerID="0fa0517827a3b5d6e9140cce54ba5a7c9af06555e255d353961e3ff4e5689f5d"
Nov 02 16:04:52.738626 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:04:52.738606    5041 scope.go:122] "RemoveContainer" containerID="ce5cb2174d828b79dbd31532ffb6d3509540ca23b751ab86d0666144dc8bb476"
Nov 02 16:05:01.044858 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:01.044817    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"]
Nov 02 16:05:01.055751 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:01.055720    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-r7mt6"]
Nov 02 16:05:01.886275 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:01.886236    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88e3d74-37ff-4c3f-a1c9-b4c51376c877" path="/var/lib/kubelet/pods/d88e3d74-37ff-4c3f-a1c9-b4c51376c877/volumes"
Nov 02 16:05:06.768916 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:05:06.768878    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:05:21.029007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:21.028967    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"]
Nov 02 16:05:21.040491 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:21.040461    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jvnwj"]
Nov 02 16:05:21.887079 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:21.887044    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ca2414-9735-4d0e-8458-0ebb43f7640e" path="/var/lib/kubelet/pods/44ca2414-9735-4d0e-8458-0ebb43f7640e/volumes"
Nov 02 16:05:36.768750 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:05:36.768661    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:05:43.030976 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:43.030943    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"]
Nov 02 16:05:43.042088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:43.042061    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ll7pw"]
Nov 02 16:05:43.885499 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:43.885464    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68919812-bfc7-4ff4-a406-bca192b71070" path="/var/lib/kubelet/pods/68919812-bfc7-4ff4-a406-bca192b71070/volumes"
Nov 02 16:05:49.441458 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:49.441417    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:05:49.456354 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:49.456327    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:05:52.792935 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:52.792897    5041 scope.go:122] "RemoveContainer" containerID="b2b0852174e5b73820b497bf055b6c2dc763c607f48a2e1dd8aac73f2c5fa03e"
Nov 02 16:05:52.802154 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:52.802132    5041 scope.go:122] "RemoveContainer" containerID="8cfe6f2265a2b4b3099641022afe63d78b9c6d6d7f5d592d048991f1ac374646"
Nov 02 16:05:52.811894 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:05:52.811872    5041 scope.go:122] "RemoveContainer" containerID="e72e9e4ebc4142523ec6f56c8c3b18246f7efa5ad7f3ac218e91f2bfa53faf7a"
Nov 02 16:06:06.766958 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:06:06.766922    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:06:13.289101 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289067    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"]
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289567    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289581    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289591    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerName="payload-processing"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289598    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerName="payload-processing"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289605    5041 cpu_manager.go:414] "containerMap: removing container" podUID="30eed75c-39fc-4e1d-96ea-8bc58db718d4" containerName="revoke-keys"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289610    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30eed75c-39fc-4e1d-96ea-8bc58db718d4" containerName="revoke-keys"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289637    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerName="payload-pre-processing"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289642    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerName="payload-pre-processing"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289652    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy"
Nov 02 16:06:13.289668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289656    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy"
Nov 02 16:06:13.289994 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289718    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9843c1f3-ff64-4ec8-b8ee-a3c512c7c07c" containerName="payload-processing"
Nov 02 16:06:13.289994 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289728    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="30eed75c-39fc-4e1d-96ea-8bc58db718d4" containerName="revoke-keys"
Nov 02 16:06:13.289994 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289738    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee348c0-6881-454d-b0c2-e5655e720c32" containerName="payload-pre-processing"
Nov 02 16:06:13.289994 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289745    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy"
Nov 02 16:06:13.289994 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.289753    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd799063-5c86-4304-8d13-ec49d350decf" containerName="istio-proxy"
Nov 02 16:06:13.293142 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.293123    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.307741 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.307720    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:06:13.307850 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.307754    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:06:13.307913 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.307849    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:06:13.319291 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.319273    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:06:13.323674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.323651    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"]
Nov 02 16:06:13.381732 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.381707    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldnr\" (UniqueName: \"kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.381878 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.381750    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.381952 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.381891    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.482789 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.482765    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldnr\" (UniqueName: \"kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.482975 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.482817    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.482975 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.482866    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.483472 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.483451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.485205 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.485185    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.506403 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.506379    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldnr\" (UniqueName: \"kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr\") pod \"maas-api-revoke-sub-d576e68b-bmmms\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.605610 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.605562    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:13.908851 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:13.908821    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"]
Nov 02 16:06:13.910173 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:06:13.910144    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7913fcda_8e62_4781_9188_982f315f8e4d.slice/crio-0daf07669c47cc3b5cf9384855118ce286b400f54a26cf7272fec52cef6b931c WatchSource:0}: Error finding container 0daf07669c47cc3b5cf9384855118ce286b400f54a26cf7272fec52cef6b931c: Status 404 returned error can't find the container with id 0daf07669c47cc3b5cf9384855118ce286b400f54a26cf7272fec52cef6b931c
Nov 02 16:06:14.224428 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:14.224383    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms" event={"ID":"7913fcda-8e62-4781-9188-982f315f8e4d","Type":"ContainerStarted","Data":"0daf07669c47cc3b5cf9384855118ce286b400f54a26cf7272fec52cef6b931c"}
Nov 02 16:06:15.231402 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:15.231354    5041 generic.go:356] "Generic (PLEG): container finished" podID="7913fcda-8e62-4781-9188-982f315f8e4d" containerID="6a2e426aab36a9700b8d833581e97115d60a81777e47a342af106c16963662a6" exitCode=0
Nov 02 16:06:15.231819 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:15.231460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms" event={"ID":"7913fcda-8e62-4781-9188-982f315f8e4d","Type":"ContainerDied","Data":"6a2e426aab36a9700b8d833581e97115d60a81777e47a342af106c16963662a6"}
Nov 02 16:06:16.381219 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.381197    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:16.408801 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.408777    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca\") pod \"7913fcda-8e62-4781-9188-982f315f8e4d\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") "
Nov 02 16:06:16.408904 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.408839    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr\" (UniqueName: \"kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr\") pod \"7913fcda-8e62-4781-9188-982f315f8e4d\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") "
Nov 02 16:06:16.408904 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.408862    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca\") pod \"7913fcda-8e62-4781-9188-982f315f8e4d\" (UID: \"7913fcda-8e62-4781-9188-982f315f8e4d\") "
Nov 02 16:06:16.409139 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.409109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca" pod "7913fcda-8e62-4781-9188-982f315f8e4d" (UID: "7913fcda-8e62-4781-9188-982f315f8e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:06:16.410780 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.410751    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr" pod "7913fcda-8e62-4781-9188-982f315f8e4d" (UID: "7913fcda-8e62-4781-9188-982f315f8e4d"). InnerVolumeSpecName "kube-api-access-4ldnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:06:16.410780 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.410771    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca" pod "7913fcda-8e62-4781-9188-982f315f8e4d" (UID: "7913fcda-8e62-4781-9188-982f315f8e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:06:16.510254 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.510189    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldnr\" (UniqueName: \"kubernetes.io/projected/7913fcda-8e62-4781-9188-982f315f8e4d-kube-api-access-4ldnr\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:06:16.510254 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.510218    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:06:16.510254 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:16.510228    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7913fcda-8e62-4781-9188-982f315f8e4d-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:06:17.245171 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:17.245139    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"
Nov 02 16:06:17.245352 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:17.245167    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms" event={"ID":"7913fcda-8e62-4781-9188-982f315f8e4d","Type":"ContainerDied","Data":"0daf07669c47cc3b5cf9384855118ce286b400f54a26cf7272fec52cef6b931c"}
Nov 02 16:06:17.245352 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:17.245239    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0daf07669c47cc3b5cf9384855118ce286b400f54a26cf7272fec52cef6b931c"
Nov 02 16:06:34.755978 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.755937    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"]
Nov 02 16:06:34.756496 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.756391    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7913fcda-8e62-4781-9188-982f315f8e4d" containerName="revoke-keys"
Nov 02 16:06:34.756496 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.756405    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7913fcda-8e62-4781-9188-982f315f8e4d" containerName="revoke-keys"
Nov 02 16:06:34.756496 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.756476    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7913fcda-8e62-4781-9188-982f315f8e4d" containerName="revoke-keys"
Nov 02 16:06:34.767344 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.767307    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.783955 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.783925    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"]
Nov 02 16:06:34.789268 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.789240    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.789424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.789326    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.789482 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.789448    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsv85\" (UniqueName: \"kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.791841 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.791814    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:06:34.791841 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.791826    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:06:34.792062 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.792005    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:06:34.794174 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.794151    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:06:34.890548 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.890520    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsv85\" (UniqueName: \"kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.890719 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.890590    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.890719 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.890635    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.891264 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.891237    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.892906 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.892883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:34.919297 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:34.919272    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsv85\" (UniqueName: \"kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85\") pod \"maas-api-revoke-sub-d2bc93bc-qjxnw\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:35.090327 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:35.090272    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:35.389304 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:35.389270    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"]
Nov 02 16:06:35.390770 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:06:35.390728    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936e1fcd_88e8_49ed_b5dd_17f769030411.slice/crio-09408ee19991a4754ddf6129213804281d88e52466e36ca35a6665a76feeada7 WatchSource:0}: Error finding container 09408ee19991a4754ddf6129213804281d88e52466e36ca35a6665a76feeada7: Status 404 returned error can't find the container with id 09408ee19991a4754ddf6129213804281d88e52466e36ca35a6665a76feeada7
Nov 02 16:06:36.386471 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:36.386429    5041 generic.go:356] "Generic (PLEG): container finished" podID="936e1fcd-88e8-49ed-b5dd-17f769030411" containerID="4bcde0e2c9b24bacc3afe2c2e4a2760f8691e74d233e5f2798f6218c83182888" exitCode=0
Nov 02 16:06:36.386880 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:36.386517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw" event={"ID":"936e1fcd-88e8-49ed-b5dd-17f769030411","Type":"ContainerDied","Data":"4bcde0e2c9b24bacc3afe2c2e4a2760f8691e74d233e5f2798f6218c83182888"}
Nov 02 16:06:36.386880 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:36.386571    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw" event={"ID":"936e1fcd-88e8-49ed-b5dd-17f769030411","Type":"ContainerStarted","Data":"09408ee19991a4754ddf6129213804281d88e52466e36ca35a6665a76feeada7"}
Nov 02 16:06:36.767552 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:06:36.767514    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:06:37.541600 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.541570    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:37.613950 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.613924    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85\" (UniqueName: \"kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85\") pod \"936e1fcd-88e8-49ed-b5dd-17f769030411\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") "
Nov 02 16:06:37.614102 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.614038    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca\") pod \"936e1fcd-88e8-49ed-b5dd-17f769030411\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") "
Nov 02 16:06:37.614177 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.614139    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca\") pod \"936e1fcd-88e8-49ed-b5dd-17f769030411\" (UID: \"936e1fcd-88e8-49ed-b5dd-17f769030411\") "
Nov 02 16:06:37.614479 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.614454    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca" pod "936e1fcd-88e8-49ed-b5dd-17f769030411" (UID: "936e1fcd-88e8-49ed-b5dd-17f769030411"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:06:37.616134 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.616101    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85" pod "936e1fcd-88e8-49ed-b5dd-17f769030411" (UID: "936e1fcd-88e8-49ed-b5dd-17f769030411"). InnerVolumeSpecName "kube-api-access-bsv85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:06:37.616245 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.616209    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca" pod "936e1fcd-88e8-49ed-b5dd-17f769030411" (UID: "936e1fcd-88e8-49ed-b5dd-17f769030411"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:06:37.715279 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.715251    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:06:37.715279 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.715278    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsv85\" (UniqueName: \"kubernetes.io/projected/936e1fcd-88e8-49ed-b5dd-17f769030411-kube-api-access-bsv85\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:06:37.715446 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:37.715292    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936e1fcd-88e8-49ed-b5dd-17f769030411-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:06:38.399547 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:38.399511    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"
Nov 02 16:06:38.399692 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:38.399552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw" event={"ID":"936e1fcd-88e8-49ed-b5dd-17f769030411","Type":"ContainerDied","Data":"09408ee19991a4754ddf6129213804281d88e52466e36ca35a6665a76feeada7"}
Nov 02 16:06:38.399692 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:38.399601    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09408ee19991a4754ddf6129213804281d88e52466e36ca35a6665a76feeada7"
Nov 02 16:06:46.958172 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:46.958137    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"]
Nov 02 16:06:46.958855 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:46.958830    5041 cpu_manager.go:414] "containerMap: removing container" podUID="936e1fcd-88e8-49ed-b5dd-17f769030411" containerName="revoke-keys"
Nov 02 16:06:46.958950 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:46.958858    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936e1fcd-88e8-49ed-b5dd-17f769030411" containerName="revoke-keys"
Nov 02 16:06:46.959010 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:46.958986    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="936e1fcd-88e8-49ed-b5dd-17f769030411" containerName="revoke-keys"
Nov 02 16:06:46.962644 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:46.962619    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:46.980128 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:46.980109    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-bbe3a8-openshift-default-dockercfg-78wq6\""
Nov 02 16:06:47.002476 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.002444    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"]
Nov 02 16:06:47.092341 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092308    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092474 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092346    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092474 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092382    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092474 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092443    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092594 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092594 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092536    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092686 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092658    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092724 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092697    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw27v\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.092724 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.092715    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193435 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193393    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193435 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193437    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193497    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193527    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw27v\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193552    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193660    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193688    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193965 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193718    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.193965 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.193754    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.194055 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.194028    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.194101 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.194040    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.194328 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.194297    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.194489 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.194304    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.194763 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.194742    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.195942 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.195913    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.196054 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.196032    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.217444 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.217357    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.217547 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.217467    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw27v\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v\") pod \"e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") " pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.277963 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.277938    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:47.592486 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.592458    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"]
Nov 02 16:06:47.594012 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:06:47.593983    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d25ccd8_3d27_49d0_80f7_59bc58293c4c.slice/crio-5013fbee9b128aec53efcc6363eec60c5f9328ee7d102b8a69f441d8c09ae258 WatchSource:0}: Error finding container 5013fbee9b128aec53efcc6363eec60c5f9328ee7d102b8a69f441d8c09ae258: Status 404 returned error can't find the container with id 5013fbee9b128aec53efcc6363eec60c5f9328ee7d102b8a69f441d8c09ae258
Nov 02 16:06:47.596111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.596073    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:06:47.596204 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.596140    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:06:47.596204 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:47.596191    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:06:48.466480 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:48.466440    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" event={"ID":"9d25ccd8-3d27-49d0-80f7-59bc58293c4c","Type":"ContainerStarted","Data":"e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"}
Nov 02 16:06:48.466866 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:48.466502    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" event={"ID":"9d25ccd8-3d27-49d0-80f7-59bc58293c4c","Type":"ContainerStarted","Data":"5013fbee9b128aec53efcc6363eec60c5f9328ee7d102b8a69f441d8c09ae258"}
Nov 02 16:06:48.523074 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:48.523026    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podStartSLOduration=2.523014024 podStartE2EDuration="2.523014024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:48.521039482 +0000 UTC m=+4260.779108932" watchObservedRunningTime="2026-11-02 16:06:48.523014024 +0000 UTC m=+4260.781083474"
Nov 02 16:06:49.278532 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:49.278498    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:49.283312 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:49.283289    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:49.473285 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:49.473254    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:49.474339 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:49.474314    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:06:52.443675 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.443633    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"]
Nov 02 16:06:52.447767 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.447743    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.462505 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.462477    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-bbe3a8-serving-cert\""
Nov 02 16:06:52.474143 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.474115    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:06:52.474280 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.474200    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cz4f2\""
Nov 02 16:06:52.474280 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.474252    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 16:06:52.474472 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.474329    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:06:52.478268 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.478239    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"]
Nov 02 16:06:52.542649 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.542613    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7z9\" (UniqueName: \"kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.542828 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.542659    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.542919 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.542882    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.542975 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.542924    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.645397 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.645340    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.645809 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.645787    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.646296 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.646275    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7z9\" (UniqueName: \"kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.646568 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:06:52.645737    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-serving-cert: secret "maas-api-e2e-worker-single-main-bbe3a8-serving-cert" not found
Nov 02 16:06:52.646659 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:06:52.646638    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls podName:b05d06ef-4b9e-4832-a67e-68749451f2ad nodeName:}" failed. No retries permitted until 2026-11-02 16:06:53.146612735 +0000 UTC m=+4265.404682166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls") pod "maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" (UID: "b05d06ef-4b9e-4832-a67e-68749451f2ad") : secret "maas-api-e2e-worker-single-main-bbe3a8-serving-cert" not found
Nov 02 16:06:52.646789 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.646760    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.647476 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.647416    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.652095 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.652073    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.674514 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.674485    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7z9\" (UniqueName: \"kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:52.843550 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.843472    5041 scope.go:122] "RemoveContainer" containerID="06c14f2e799828e000c168644efd1704fd89f2ada7eec4eca8c1fb4d0e423d5e"
Nov 02 16:06:52.853282 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.853261    5041 scope.go:122] "RemoveContainer" containerID="59a1461105e6d89b2db957ba75d33737c9d4008e678b3aae2c6fc6adb9e4f1bf"
Nov 02 16:06:52.863215 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.863191    5041 scope.go:122] "RemoveContainer" containerID="8d427649a48b7c9f848246ba965889712b50ae3ac1b3c3b367e849f513f66cc9"
Nov 02 16:06:52.871714 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.871690    5041 scope.go:122] "RemoveContainer" containerID="62dbabe8990ff435431d7906d228df80fdb98733726293d21267a501d77b1750"
Nov 02 16:06:52.945974 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.945943    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"]
Nov 02 16:06:52.951243 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.951219    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:52.966922 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.966895    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4ae89ce0-dockercfg-7rt8b\""
Nov 02 16:06:52.966922 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.966903    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bbe3a8\""
Nov 02 16:06:52.976615 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:52.976593    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"]
Nov 02 16:06:53.003148 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.003125    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"]
Nov 02 16:06:53.039791 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.039657    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.058864 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.051001    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvhph\" (UniqueName: \"kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph\") pod \"payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.058864 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.051097    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.068417 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.064358    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"]
Nov 02 16:06:53.072642 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.070400    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"]
Nov 02 16:06:53.074108 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.074081    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b2rvt"]
Nov 02 16:06:53.152291 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.152213    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpt49\" (UniqueName: \"kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49\") pod \"payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.152291 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.152267    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvhph\" (UniqueName: \"kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph\") pod \"payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.152522 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.152330    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.152522 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.152427    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.152657 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.152571    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:53.152725 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:06:53.152675    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-serving-cert: secret "maas-api-e2e-worker-single-main-bbe3a8-serving-cert" not found
Nov 02 16:06:53.152789 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:06:53.152733    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls podName:b05d06ef-4b9e-4832-a67e-68749451f2ad nodeName:}" failed. No retries permitted until 2026-11-02 16:06:54.152720046 +0000 UTC m=+4266.410789473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls") pod "maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" (UID: "b05d06ef-4b9e-4832-a67e-68749451f2ad") : secret "maas-api-e2e-worker-single-main-bbe3a8-serving-cert" not found
Nov 02 16:06:53.153214 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.153191    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.176108 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.176084    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvhph\" (UniqueName: \"kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph\") pod \"payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.253762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.253726    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpt49\" (UniqueName: \"kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49\") pod \"payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.253910 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.253885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.254595 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.254573    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.265559 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.265535    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:53.279670 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.279618    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpt49\" (UniqueName: \"kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49\") pod \"payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.379940 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.379902    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:53.565780 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.565747    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"]
Nov 02 16:06:53.566779 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:06:53.566747    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f214c14_74b0_4600_9208_a0de09a91a48.slice/crio-d78c5dc77978987ded408349d9bf869ec2c255b8bda5cfce41f772484e213905 WatchSource:0}: Error finding container d78c5dc77978987ded408349d9bf869ec2c255b8bda5cfce41f772484e213905: Status 404 returned error can't find the container with id d78c5dc77978987ded408349d9bf869ec2c255b8bda5cfce41f772484e213905
Nov 02 16:06:53.696386 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.696335    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"]
Nov 02 16:06:53.698162 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:06:53.698134    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0fd8ed_246a_4305_9d44_fb6a2b2cf41c.slice/crio-61ecd47020345fc0f75c5f18b75c455b62be8798844dd5c18bcba7deaba063a1 WatchSource:0}: Error finding container 61ecd47020345fc0f75c5f18b75c455b62be8798844dd5c18bcba7deaba063a1: Status 404 returned error can't find the container with id 61ecd47020345fc0f75c5f18b75c455b62be8798844dd5c18bcba7deaba063a1
Nov 02 16:06:53.887319 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:53.887290    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101cfa8d-0553-4180-94e7-c9ba2ca41497" path="/var/lib/kubelet/pods/101cfa8d-0553-4180-94e7-c9ba2ca41497/volumes"
Nov 02 16:06:54.165653 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.165567    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:54.167843 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.167817    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:54.261571 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.261549    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:54.504321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.504277    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" event={"ID":"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c","Type":"ContainerStarted","Data":"53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f"}
Nov 02 16:06:54.504561 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.504396    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:06:54.504561 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.504435    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" event={"ID":"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c","Type":"ContainerStarted","Data":"61ecd47020345fc0f75c5f18b75c455b62be8798844dd5c18bcba7deaba063a1"}
Nov 02 16:06:54.505850 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.505815    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" event={"ID":"5f214c14-74b0-4600-9208-a0de09a91a48","Type":"ContainerStarted","Data":"75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194"}
Nov 02 16:06:54.505850 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.505854    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" event={"ID":"5f214c14-74b0-4600-9208-a0de09a91a48","Type":"ContainerStarted","Data":"d78c5dc77978987ded408349d9bf869ec2c255b8bda5cfce41f772484e213905"}
Nov 02 16:06:54.506007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.505872    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:06:54.545820 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.545774    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" podStartSLOduration=2.5457620260000002 podStartE2EDuration="2.545762026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:54.542868344 +0000 UTC m=+4266.800937796" watchObservedRunningTime="2026-11-02 16:06:54.545762026 +0000 UTC m=+4266.803831476"
Nov 02 16:06:54.583029 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.583003    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"]
Nov 02 16:06:54.584050 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:54.584004    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" podStartSLOduration=2.583989566 podStartE2EDuration="2.583989566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:54.582158163 +0000 UTC m=+4266.840227612" watchObservedRunningTime="2026-11-02 16:06:54.583989566 +0000 UTC m=+4266.842059018"
Nov 02 16:06:54.584497 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:06:54.584471    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05d06ef_4b9e_4832_a67e_68749451f2ad.slice/crio-aa44e018fef6d5f421e2dae58235177d7b2bba503535c6f0b68dfebeb95076e7 WatchSource:0}: Error finding container aa44e018fef6d5f421e2dae58235177d7b2bba503535c6f0b68dfebeb95076e7: Status 404 returned error can't find the container with id aa44e018fef6d5f421e2dae58235177d7b2bba503535c6f0b68dfebeb95076e7
Nov 02 16:06:55.513832 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:55.513793    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" event={"ID":"b05d06ef-4b9e-4832-a67e-68749451f2ad","Type":"ContainerStarted","Data":"0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c"}
Nov 02 16:06:55.514002 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:55.513868    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" event={"ID":"b05d06ef-4b9e-4832-a67e-68749451f2ad","Type":"ContainerStarted","Data":"aa44e018fef6d5f421e2dae58235177d7b2bba503535c6f0b68dfebeb95076e7"}
Nov 02 16:06:55.514097 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:55.514005    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:06:55.554685 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:06:55.554640    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" podStartSLOduration=3.091123788 podStartE2EDuration="3.554627667s" totalImagesPullingTime="463.503879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:55.554218687 +0000 UTC m=+4267.812288137" watchObservedRunningTime="2026-11-02 16:06:55.554627667 +0000 UTC m=+4267.812697117"
Nov 02 16:07:01.541176 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:01.541085    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:07:04.038743 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:04.038708    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"]
Nov 02 16:07:04.049710 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:04.049685    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-6dcfx"]
Nov 02 16:07:05.393053 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.393000    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": read tcp 10.132.0.2:46310->10.132.1.96:15021: read: connection reset by peer" start-of-body=
Nov 02 16:07:05.393774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.393091    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": read tcp 10.132.0.2:46310->10.132.1.96:15021: read: connection reset by peer"
Nov 02 16:07:05.516828 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.516805    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:07:05.516962 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.516867    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:07:05.592509 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.592434    5041 generic.go:356] "Generic (PLEG): container finished" podID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerID="e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d" exitCode=0
Nov 02 16:07:05.592654 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.592511    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" event={"ID":"9d25ccd8-3d27-49d0-80f7-59bc58293c4c","Type":"ContainerDied","Data":"e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"}
Nov 02 16:07:05.592983 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.592965    5041 scope.go:122] "RemoveContainer" containerID="e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"
Nov 02 16:07:05.594333 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.594298    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:07:05.594441 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.594398    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:07:05.594484 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.594445    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:07:05.889359 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:05.889318    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0759f3a-1a6b-4c32-bb13-5b234b007117" path="/var/lib/kubelet/pods/d0759f3a-1a6b-4c32-bb13-5b234b007117/volumes"
Nov 02 16:07:06.601189 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:06.601150    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" event={"ID":"9d25ccd8-3d27-49d0-80f7-59bc58293c4c","Type":"ContainerStarted","Data":"f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21"}
Nov 02 16:07:06.769294 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:07:06.769257    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:07:07.278991 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:07.278957    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:07:07.279224 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:07.279081    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 16:07:07.279224 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:07.279127    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 16:07:08.279276 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:08.279245    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 16:07:08.279714 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:08.279304    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 16:07:09.127622 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.127585    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"]
Nov 02 16:07:09.156230 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.156202    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.166227 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.166196    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"]
Nov 02 16:07:09.171498 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.171479    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bbe3a8\"/\"kube-root-ca.crt\""
Nov 02 16:07:09.182516 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.182489    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bbe3a8\"/\"default-dockercfg-fhj4b\""
Nov 02 16:07:09.182701 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.182679    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bbe3a8\"/\"simulator-single-main-bbe3a8-kserve-self-signed-certs\""
Nov 02 16:07:09.182701 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.182688    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bbe3a8\"/\"openshift-service-ca.crt\""
Nov 02 16:07:09.263616 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.263583    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.263616 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.263617    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.263807 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.263634    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.263864 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.263843    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.263904 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.263873    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.263904 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.263895    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq9m\" (UniqueName: \"kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.278689 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.278660    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 16:07:09.278805 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.278710    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 16:07:09.365046 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365006    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.365046 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365049    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.365545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365069    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq9m\" (UniqueName: \"kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.365545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365109    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.365545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365126    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.365545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365142    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.366005 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.365978    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.366280 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.366260    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.366419 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.366393    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.367683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.367659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.368931 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.368902    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.394039 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.393976    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq9m\" (UniqueName: \"kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m\") pod \"simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") " pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.473063 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.473032    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:09.808764 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:09.808725    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"]
Nov 02 16:07:09.809210 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:07:09.809182    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c362e59_e439_46a8_9201_01eb82dc58f1.slice/crio-0424a0485a83beb49505c179bb0c4ead5974bae69ae8e1753ea39fc76d4e8a2f WatchSource:0}: Error finding container 0424a0485a83beb49505c179bb0c4ead5974bae69ae8e1753ea39fc76d4e8a2f: Status 404 returned error can't find the container with id 0424a0485a83beb49505c179bb0c4ead5974bae69ae8e1753ea39fc76d4e8a2f
Nov 02 16:07:10.279109 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:10.279062    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 16:07:10.279321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:10.279148    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 16:07:10.655489 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:10.655396    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" event={"ID":"4c362e59-e439-46a8-9201-01eb82dc58f1","Type":"ContainerStarted","Data":"97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe"}
Nov 02 16:07:10.655489 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:10.655450    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" event={"ID":"4c362e59-e439-46a8-9201-01eb82dc58f1","Type":"ContainerStarted","Data":"0424a0485a83beb49505c179bb0c4ead5974bae69ae8e1753ea39fc76d4e8a2f"}
Nov 02 16:07:10.698101 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:10.698050    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" podStartSLOduration=1.698036613 podStartE2EDuration="1.698036613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:07:10.694850069 +0000 UTC m=+4282.952919519" watchObservedRunningTime="2026-11-02 16:07:10.698036613 +0000 UTC m=+4282.956106097"
Nov 02 16:07:11.278619 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:11.278574    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 16:07:11.278825 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:11.278631    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 16:07:11.473187 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:11.473150    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:12.282984 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:12.282945    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:07:12.283803 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:12.283769    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:07:12.284468 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:12.284448    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:07:19.473427 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:19.473359    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:19.485845 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:19.485820    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:19.720834 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:19.720805    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:07:36.769079 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:07:36.769038    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:07:52.911940 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:52.911906    5041 scope.go:122] "RemoveContainer" containerID="6cba6a115429cb914ad12082eaa439f30e2b0f2137a1383b5657bb445928f68a"
Nov 02 16:07:52.921604 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:52.921580    5041 scope.go:122] "RemoveContainer" containerID="15b6aae0caa0391a35f53e1e4b1633e3ea8c14905f78fb1bb89261bcef167a47"
Nov 02 16:07:56.445018 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:07:56.444980    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"]
Nov 02 16:08:02.127125 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:02.127084    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"]
Nov 02 16:08:02.163669 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:02.163634    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"]
Nov 02 16:08:04.430106 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.430066    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"]
Nov 02 16:08:04.430545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.430270    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" podUID="b05d06ef-4b9e-4832-a67e-68749451f2ad" containerName="maas-api" containerID="cri-o://0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c" gracePeriod=30
Nov 02 16:08:04.910044 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.910015    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:08:04.982093 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.982018    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"
Nov 02 16:08:04.982093 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.982040    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" event={"ID":"b05d06ef-4b9e-4832-a67e-68749451f2ad","Type":"ContainerDied","Data":"0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c"}
Nov 02 16:08:04.982331 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.982103    5041 scope.go:122] "RemoveContainer" containerID="0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c"
Nov 02 16:08:04.982331 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.982006    5041 generic.go:356] "Generic (PLEG): container finished" podID="b05d06ef-4b9e-4832-a67e-68749451f2ad" containerID="0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c" exitCode=0
Nov 02 16:08:04.982331 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.982293    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls" event={"ID":"b05d06ef-4b9e-4832-a67e-68749451f2ad","Type":"ContainerDied","Data":"aa44e018fef6d5f421e2dae58235177d7b2bba503535c6f0b68dfebeb95076e7"}
Nov 02 16:08:04.995867 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.995844    5041 scope.go:122] "RemoveContainer" containerID="0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c"
Nov 02 16:08:04.996196 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:04.996168    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c\": container with ID starting with 0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c not found: ID does not exist" containerID="0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c"
Nov 02 16:08:04.996261 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:04.996204    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c"} err="failed to get container status \"0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c\": rpc error: code = NotFound desc = could not find container \"0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c\": container with ID starting with 0b2e78919d6d5d1b754ea678255bb58a25c52cbdc1fb6779f61d04e74160c06c not found: ID does not exist"
Nov 02 16:08:05.018209 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.018183    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca\") pod \"b05d06ef-4b9e-4832-a67e-68749451f2ad\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") "
Nov 02 16:08:05.018351 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.018228    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9\" (UniqueName: \"kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9\") pod \"b05d06ef-4b9e-4832-a67e-68749451f2ad\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") "
Nov 02 16:08:05.018351 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.018319    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs\") pod \"b05d06ef-4b9e-4832-a67e-68749451f2ad\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") "
Nov 02 16:08:05.018523 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.018400    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") pod \"b05d06ef-4b9e-4832-a67e-68749451f2ad\" (UID: \"b05d06ef-4b9e-4832-a67e-68749451f2ad\") "
Nov 02 16:08:05.018633 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.018610    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca" pod "b05d06ef-4b9e-4832-a67e-68749451f2ad" (UID: "b05d06ef-4b9e-4832-a67e-68749451f2ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:08:05.018771 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.018754    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05d06ef-4b9e-4832-a67e-68749451f2ad-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:05.020529 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.020503    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs" pod "b05d06ef-4b9e-4832-a67e-68749451f2ad" (UID: "b05d06ef-4b9e-4832-a67e-68749451f2ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:08:05.020641 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.020560    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls" pod "b05d06ef-4b9e-4832-a67e-68749451f2ad" (UID: "b05d06ef-4b9e-4832-a67e-68749451f2ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:08:05.020693 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.020657    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9" pod "b05d06ef-4b9e-4832-a67e-68749451f2ad" (UID: "b05d06ef-4b9e-4832-a67e-68749451f2ad"). InnerVolumeSpecName "kube-api-access-7f7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:08:05.119993 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.119952    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f7z9\" (UniqueName: \"kubernetes.io/projected/b05d06ef-4b9e-4832-a67e-68749451f2ad-kube-api-access-7f7z9\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:05.119993 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.119986    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:05.120220 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.120020    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05d06ef-4b9e-4832-a67e-68749451f2ad-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:05.327351 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.327317    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"]
Nov 02 16:08:05.338441 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.338411    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbe3a8-799c48b8ff-562ls"]
Nov 02 16:08:05.887185 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:05.887150    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05d06ef-4b9e-4832-a67e-68749451f2ad" path="/var/lib/kubelet/pods/b05d06ef-4b9e-4832-a67e-68749451f2ad/volumes"
Nov 02 16:08:06.769051 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:06.769014    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:08:07.128487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:07.128389    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerName="payload-processing" containerID="cri-o://53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f" gracePeriod=25
Nov 02 16:08:07.164900 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:07.164859    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" containerName="payload-pre-processing" containerID="cri-o://75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194" gracePeriod=25
Nov 02 16:08:11.467533 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.467486    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" podUID="4c362e59-e439-46a8-9201-01eb82dc58f1" containerName="main" containerID="cri-o://97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe" gracePeriod=45
Nov 02 16:08:11.725164 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.725106    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:08:11.888745 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.888700    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache\") pod \"4c362e59-e439-46a8-9201-01eb82dc58f1\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") "
Nov 02 16:08:11.888939 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.888845    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m\" (UniqueName: \"kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m\") pod \"4c362e59-e439-46a8-9201-01eb82dc58f1\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") "
Nov 02 16:08:11.888939 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.888880    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm\") pod \"4c362e59-e439-46a8-9201-01eb82dc58f1\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") "
Nov 02 16:08:11.888939 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.888913    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home\") pod \"4c362e59-e439-46a8-9201-01eb82dc58f1\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") "
Nov 02 16:08:11.889131 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.888967    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs\") pod \"4c362e59-e439-46a8-9201-01eb82dc58f1\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") "
Nov 02 16:08:11.889131 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.888995    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache" pod "4c362e59-e439-46a8-9201-01eb82dc58f1" (UID: "4c362e59-e439-46a8-9201-01eb82dc58f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:11.889131 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.889059    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir\") pod \"4c362e59-e439-46a8-9201-01eb82dc58f1\" (UID: \"4c362e59-e439-46a8-9201-01eb82dc58f1\") "
Nov 02 16:08:11.889468 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.889447    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:11.889592 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.889466    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home" pod "4c362e59-e439-46a8-9201-01eb82dc58f1" (UID: "4c362e59-e439-46a8-9201-01eb82dc58f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:11.889713 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.889691    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir" pod "4c362e59-e439-46a8-9201-01eb82dc58f1" (UID: "4c362e59-e439-46a8-9201-01eb82dc58f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:11.891632 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.891601    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs" pod "4c362e59-e439-46a8-9201-01eb82dc58f1" (UID: "4c362e59-e439-46a8-9201-01eb82dc58f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:08:11.891736 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.891657    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm" pod "4c362e59-e439-46a8-9201-01eb82dc58f1" (UID: "4c362e59-e439-46a8-9201-01eb82dc58f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:11.891736 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.891683    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m" pod "4c362e59-e439-46a8-9201-01eb82dc58f1" (UID: "4c362e59-e439-46a8-9201-01eb82dc58f1"). InnerVolumeSpecName "kube-api-access-qqq9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:08:11.990610 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.990512    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq9m\" (UniqueName: \"kubernetes.io/projected/4c362e59-e439-46a8-9201-01eb82dc58f1-kube-api-access-qqq9m\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:11.990610 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.990552    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:11.990610 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.990568    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:11.990610 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.990585    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c362e59-e439-46a8-9201-01eb82dc58f1-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:11.990610 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:11.990600    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c362e59-e439-46a8-9201-01eb82dc58f1-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:12.031228 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.031188    5041 generic.go:356] "Generic (PLEG): container finished" podID="4c362e59-e439-46a8-9201-01eb82dc58f1" containerID="97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe" exitCode=0
Nov 02 16:08:12.031707 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.031670    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" event={"ID":"4c362e59-e439-46a8-9201-01eb82dc58f1","Type":"ContainerDied","Data":"97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe"}
Nov 02 16:08:12.031874 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.031857    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" event={"ID":"4c362e59-e439-46a8-9201-01eb82dc58f1","Type":"ContainerDied","Data":"0424a0485a83beb49505c179bb0c4ead5974bae69ae8e1753ea39fc76d4e8a2f"}
Nov 02 16:08:12.032009 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.031996    5041 scope.go:122] "RemoveContainer" containerID="97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe"
Nov 02 16:08:12.032656 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.032635    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"
Nov 02 16:08:12.033004 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.032702    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"]
Nov 02 16:08:12.033256 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.033200    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" containerID="cri-o://f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21" gracePeriod=30
Nov 02 16:08:12.055250 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.055224    5041 scope.go:122] "RemoveContainer" containerID="97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe"
Nov 02 16:08:12.055821 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:12.055779    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe\": container with ID starting with 97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe not found: ID does not exist" containerID="97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe"
Nov 02 16:08:12.055931 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.055831    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe"} err="failed to get container status \"97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe\": rpc error: code = NotFound desc = could not find container \"97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe\": container with ID starting with 97eb366ffce15012498f842f4b54b0aefc7a8940eebfbd2629ed7c3a2e110dbe not found: ID does not exist"
Nov 02 16:08:12.133390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.131008    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"]
Nov 02 16:08:12.134896 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.134858    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx"]
Nov 02 16:08:12.286207 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.286122    5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 16:08:12.286207 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.286180    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 16:08:12.710481 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:12.710436    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-bbe3a8/simulator-single-main-bbe3a8-kserve-fc8545895-c5bbx" podUID="4c362e59-e439-46a8-9201-01eb82dc58f1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:08:13.886461 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:13.886427    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c362e59-e439-46a8-9201-01eb82dc58f1" path="/var/lib/kubelet/pods/4c362e59-e439-46a8-9201-01eb82dc58f1/volumes"
Nov 02 16:08:15.515496 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:15.515455    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:08:15.515496 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:15.515484    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:08:17.313628 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.313566    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:08:17.450107 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450071    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450316 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450117    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450316 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450169    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450316 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450209    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450316 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450243    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450316 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450304    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450653 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450392    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450653 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450653 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450451    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs\") pod \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\" (UID: \"9d25ccd8-3d27-49d0-80f7-59bc58293c4c\") "
Nov 02 16:08:17.450653 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450570    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:17.450653 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450598    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:17.450936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450642    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:17.450936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450824    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:17.450936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450860    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:08:17.450936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450909    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.451092 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450939    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.451092 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450957    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.451092 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.450969    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.452589 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.452557    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "kube-api-access-zw27v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:08:17.452697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.452587    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:08:17.452697 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.452594    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 16:08:17.452830 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.452807    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy" pod "9d25ccd8-3d27-49d0-80f7-59bc58293c4c" (UID: "9d25ccd8-3d27-49d0-80f7-59bc58293c4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:08:17.552567 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.552526    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw27v\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-kube-api-access-zw27v\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.552567 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.552563    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.552567 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.552573    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.552810 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.552581    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:17.552810 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:17.552589    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25ccd8-3d27-49d0-80f7-59bc58293c4c-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:18.079821 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.079784    5041 generic.go:356] "Generic (PLEG): container finished" podID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerID="f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21" exitCode=0
Nov 02 16:08:18.080035 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.079844    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"
Nov 02 16:08:18.080035 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.079873    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" event={"ID":"9d25ccd8-3d27-49d0-80f7-59bc58293c4c","Type":"ContainerDied","Data":"f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21"}
Nov 02 16:08:18.080035 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.079931    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd" event={"ID":"9d25ccd8-3d27-49d0-80f7-59bc58293c4c","Type":"ContainerDied","Data":"5013fbee9b128aec53efcc6363eec60c5f9328ee7d102b8a69f441d8c09ae258"}
Nov 02 16:08:18.080035 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.079956    5041 scope.go:122] "RemoveContainer" containerID="f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21"
Nov 02 16:08:18.089255 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.089236    5041 scope.go:122] "RemoveContainer" containerID="e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"
Nov 02 16:08:18.097838 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.097811    5041 scope.go:122] "RemoveContainer" containerID="f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21"
Nov 02 16:08:18.098087 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:18.098067    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21\": container with ID starting with f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21 not found: ID does not exist" containerID="f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21"
Nov 02 16:08:18.098164 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.098099    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21"} err="failed to get container status \"f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21\": rpc error: code = NotFound desc = could not find container \"f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21\": container with ID starting with f9a46e3778af20260f1a9b1e1fdbdf38a3cf4515e2e8bb8006d7d8c8f03b7b21 not found: ID does not exist"
Nov 02 16:08:18.098164 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.098123    5041 scope.go:122] "RemoveContainer" containerID="e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"
Nov 02 16:08:18.098415 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:18.098385    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d\": container with ID starting with e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d not found: ID does not exist" containerID="e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"
Nov 02 16:08:18.098521 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.098415    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d"} err="failed to get container status \"e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d\": rpc error: code = NotFound desc = could not find container \"e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d\": container with ID starting with e4abf437f7d57f3cdc8878dafddd5bd8c19a7af0e516c9f10f70d017e41e2a2d not found: ID does not exist"
Nov 02 16:08:18.142748 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.142715    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"]
Nov 02 16:08:18.154096 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:18.154033    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbe3a8-openshift-default-6c46c556b97n5kd"]
Nov 02 16:08:19.886992 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:19.886955    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" path="/var/lib/kubelet/pods/9d25ccd8-3d27-49d0-80f7-59bc58293c4c/volumes"
Nov 02 16:08:25.515528 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:25.515489    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:08:25.515946 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:25.515489    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 16:08:26.375001 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.374967    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"]
Nov 02 16:08:26.375451 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375433    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4c362e59-e439-46a8-9201-01eb82dc58f1" containerName="main"
Nov 02 16:08:26.375451 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375452    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c362e59-e439-46a8-9201-01eb82dc58f1" containerName="main"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375471    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b05d06ef-4b9e-4832-a67e-68749451f2ad" containerName="maas-api"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375477    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05d06ef-4b9e-4832-a67e-68749451f2ad" containerName="maas-api"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375489    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375495    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375581    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c362e59-e439-46a8-9201-01eb82dc58f1" containerName="main"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375591    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05d06ef-4b9e-4832-a67e-68749451f2ad" containerName="maas-api"
Nov 02 16:08:26.375631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.375600    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy"
Nov 02 16:08:26.380558 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.380533    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.406423 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.406394    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1beeeb25-kserve-self-signed-certs\""
Nov 02 16:08:26.409488 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.409462    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"]
Nov 02 16:08:26.431310 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.431279    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.431454 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.431339    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.431523 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.431493    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggcsx\" (UniqueName: \"kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.431564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.431551    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.431658 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.431638    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.431702 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.431671    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.532401 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532346    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggcsx\" (UniqueName: \"kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.532772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532410    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.532772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532454    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.532772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.532772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.532772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532547    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.533069 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532919    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.533069 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.532928    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.533069 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.533012    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.534722 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.534697    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.535122 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.535105    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.554772 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.554746    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggcsx\" (UniqueName: \"kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx\") pod \"e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") " pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.695617 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.695582    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:26.994746 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:26.994721    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"]
Nov 02 16:08:26.996296 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:08:26.996271    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f6a288b_c51f_4a05_ab3e_6c4396127123.slice/crio-4582e7de83c0a9428e8a73f222cebc859183c311e08cb487b90c40901ebc9158 WatchSource:0}: Error finding container 4582e7de83c0a9428e8a73f222cebc859183c311e08cb487b90c40901ebc9158: Status 404 returned error can't find the container with id 4582e7de83c0a9428e8a73f222cebc859183c311e08cb487b90c40901ebc9158
Nov 02 16:08:27.139497 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.139460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" event={"ID":"2f6a288b-c51f-4a05-ab3e-6c4396127123","Type":"ContainerStarted","Data":"26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef"}
Nov 02 16:08:27.139640 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.139509    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" event={"ID":"2f6a288b-c51f-4a05-ab3e-6c4396127123","Type":"ContainerStarted","Data":"4582e7de83c0a9428e8a73f222cebc859183c311e08cb487b90c40901ebc9158"}
Nov 02 16:08:27.182052 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.181913    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" podStartSLOduration=1.181896853 podStartE2EDuration="1.181896853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:08:27.179661331 +0000 UTC m=+4359.437730782" watchObservedRunningTime="2026-11-02 16:08:27.181896853 +0000 UTC m=+4359.439966305"
Nov 02 16:08:27.407020 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.406991    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:08:27.440923 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.440899    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:08:27.442096 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.442073    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume\") pod \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") "
Nov 02 16:08:27.442201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.442130    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49\" (UniqueName: \"kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49\") pod \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\" (UID: \"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c\") "
Nov 02 16:08:27.442453 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.442433    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume" pod "ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" (UID: "ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:08:27.444036 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.444016    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49" pod "ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" (UID: "ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c"). InnerVolumeSpecName "kube-api-access-bpt49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:08:27.542799 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.542727    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph\" (UniqueName: \"kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph\") pod \"5f214c14-74b0-4600-9208-a0de09a91a48\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") "
Nov 02 16:08:27.542799 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.542771    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume\") pod \"5f214c14-74b0-4600-9208-a0de09a91a48\" (UID: \"5f214c14-74b0-4600-9208-a0de09a91a48\") "
Nov 02 16:08:27.543130 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.542975    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:27.543130 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.542987    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpt49\" (UniqueName: \"kubernetes.io/projected/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c-kube-api-access-bpt49\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:27.543256 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.543126    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume" pod "5f214c14-74b0-4600-9208-a0de09a91a48" (UID: "5f214c14-74b0-4600-9208-a0de09a91a48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:08:27.545170 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.545143    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph" pod "5f214c14-74b0-4600-9208-a0de09a91a48" (UID: "5f214c14-74b0-4600-9208-a0de09a91a48"). InnerVolumeSpecName "kube-api-access-nvhph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:08:27.644106 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.644075    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvhph\" (UniqueName: \"kubernetes.io/projected/5f214c14-74b0-4600-9208-a0de09a91a48-kube-api-access-nvhph\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:27.644106 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.644103    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f214c14-74b0-4600-9208-a0de09a91a48-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:08:27.696336 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:27.696305    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:28.146914 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.146832    5041 generic.go:356] "Generic (PLEG): container finished" podID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerID="53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f" exitCode=0
Nov 02 16:08:28.146914 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.146880    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"
Nov 02 16:08:28.147130 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.146920    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" event={"ID":"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c","Type":"ContainerDied","Data":"53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f"}
Nov 02 16:08:28.147130 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.146973    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr" event={"ID":"ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c","Type":"ContainerDied","Data":"61ecd47020345fc0f75c5f18b75c455b62be8798844dd5c18bcba7deaba063a1"}
Nov 02 16:08:28.147130 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.147003    5041 scope.go:122] "RemoveContainer" containerID="53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f"
Nov 02 16:08:28.148507 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.148483    5041 generic.go:356] "Generic (PLEG): container finished" podID="5f214c14-74b0-4600-9208-a0de09a91a48" containerID="75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194" exitCode=0
Nov 02 16:08:28.148593 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.148527    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"
Nov 02 16:08:28.148593 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.148546    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" event={"ID":"5f214c14-74b0-4600-9208-a0de09a91a48","Type":"ContainerDied","Data":"75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194"}
Nov 02 16:08:28.148707 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.148600    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl" event={"ID":"5f214c14-74b0-4600-9208-a0de09a91a48","Type":"ContainerDied","Data":"d78c5dc77978987ded408349d9bf869ec2c255b8bda5cfce41f772484e213905"}
Nov 02 16:08:28.157615 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.157575    5041 scope.go:122] "RemoveContainer" containerID="53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f"
Nov 02 16:08:28.157845 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:28.157828    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f\": container with ID starting with 53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f not found: ID does not exist" containerID="53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f"
Nov 02 16:08:28.157905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.157852    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f"} err="failed to get container status \"53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f\": rpc error: code = NotFound desc = could not find container \"53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f\": container with ID starting with 53e4e32a61d8d392703b32237cff19c6247c9d72542ffbc8679b08c63c39b34f not found: ID does not exist"
Nov 02 16:08:28.157905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.157869    5041 scope.go:122] "RemoveContainer" containerID="75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194"
Nov 02 16:08:28.166152 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.166128    5041 scope.go:122] "RemoveContainer" containerID="75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194"
Nov 02 16:08:28.166447 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:28.166425    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194\": container with ID starting with 75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194 not found: ID does not exist" containerID="75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194"
Nov 02 16:08:28.166547 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.166453    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194"} err="failed to get container status \"75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194\": rpc error: code = NotFound desc = could not find container \"75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194\": container with ID starting with 75da6ca1ab4ae0f077302d37580d4cc356f3f2b4de0a2b7f000eade908805194 not found: ID does not exist"
Nov 02 16:08:28.196305 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.196281    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"]
Nov 02 16:08:28.207246 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.207219    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbe3a8-87fdc99b5l8vbr"]
Nov 02 16:08:28.232300 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.232281    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"]
Nov 02 16:08:28.235878 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:28.235859    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbe3a8-56965zlctl"]
Nov 02 16:08:29.891277 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:29.891176    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" path="/var/lib/kubelet/pods/5f214c14-74b0-4600-9208-a0de09a91a48/volumes"
Nov 02 16:08:29.891791 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:29.891745    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" path="/var/lib/kubelet/pods/ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c/volumes"
Nov 02 16:08:36.696435 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:36.696398    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:36.709025 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:36.708997    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:36.768273 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:08:36.768240    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:08:37.283003 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:37.282971    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:08:43.123839 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.123803    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"]
Nov 02 16:08:43.124324 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124294    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerName="payload-processing"
Nov 02 16:08:43.124324 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124320    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerName="payload-processing"
Nov 02 16:08:43.124564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124393    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" containerName="payload-pre-processing"
Nov 02 16:08:43.124564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124404    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" containerName="payload-pre-processing"
Nov 02 16:08:43.124564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124418    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy"
Nov 02 16:08:43.124564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124427    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy"
Nov 02 16:08:43.124780 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124568    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f214c14-74b0-4600-9208-a0de09a91a48" containerName="payload-pre-processing"
Nov 02 16:08:43.124780 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124582    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d25ccd8-3d27-49d0-80f7-59bc58293c4c" containerName="istio-proxy"
Nov 02 16:08:43.124780 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.124599    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0fd8ed-246a-4305-9d44-fb6a2b2cf41c" containerName="payload-processing"
Nov 02 16:08:43.129872 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.129848    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.149645 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.149618    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"]
Nov 02 16:08:43.158235 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.158210    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1beeeb25-kserve-self-signed-certs\""
Nov 02 16:08:43.241040 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.241013    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.241151 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.241071    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrh5x\" (UniqueName: \"kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.241217 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.241201    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.241260 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.241237    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.241307 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.241272    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.241418 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.241392    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.343437 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.343371    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.343827 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.343776    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrh5x\" (UniqueName: \"kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.344060 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.344017    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.344292 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.344191    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.344292 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.344262    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.345064 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.344526    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.351270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.351243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.351454 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.351427    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.351607 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.351283    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.352915 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.352883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.353407 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.353355    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.374106 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.374050    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrh5x\" (UniqueName: \"kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x\") pod \"e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") " pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.443431 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.443405    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:43.747074 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:43.747039    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"]
Nov 02 16:08:43.748493 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:08:43.748463    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73ae991f_b722_49fb_8f40_f2c9f2c99fc1.slice/crio-31a613e0226a160a06fb8266e8e10e42933dd0e745c0a068417b9b5324ff031f WatchSource:0}: Error finding container 31a613e0226a160a06fb8266e8e10e42933dd0e745c0a068417b9b5324ff031f: Status 404 returned error can't find the container with id 31a613e0226a160a06fb8266e8e10e42933dd0e745c0a068417b9b5324ff031f
Nov 02 16:08:44.314340 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:44.314301    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" event={"ID":"73ae991f-b722-49fb-8f40-f2c9f2c99fc1","Type":"ContainerStarted","Data":"1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22"}
Nov 02 16:08:44.314724 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:44.314354    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" event={"ID":"73ae991f-b722-49fb-8f40-f2c9f2c99fc1","Type":"ContainerStarted","Data":"31a613e0226a160a06fb8266e8e10e42933dd0e745c0a068417b9b5324ff031f"}
Nov 02 16:08:44.357590 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:44.357547    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" podStartSLOduration=1.357534402 podStartE2EDuration="1.357534402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:08:44.354480533 +0000 UTC m=+4376.612549983" watchObservedRunningTime="2026-11-02 16:08:44.357534402 +0000 UTC m=+4376.615603843"
Nov 02 16:08:44.443910 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:44.443886    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:52.947607 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:52.947567    5041 scope.go:122] "RemoveContainer" containerID="13fffb0d65ee6343d8ca029127d86c6fa6738a69952ee665e0071ca45ca744a0"
Nov 02 16:08:52.956865 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:52.956843    5041 scope.go:122] "RemoveContainer" containerID="f5c917c4bc5719daa053ea5299ef7541e712a248f03171558ff40ee5fcc9b401"
Nov 02 16:08:53.444155 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:53.444124    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:53.456411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:53.456389    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:54.430033 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:54.430000    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:08:59.932933 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:59.932901    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"]
Nov 02 16:08:59.936720 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:59.936698    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:08:59.952045 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:59.952028    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:08:59.952281 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:59.952260    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:08:59.967390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:08:59.967347    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"]
Nov 02 16:09:00.000235 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.000210    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:09:00.000334 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.000234    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:09:00.066759 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.066731    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.066882 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.066778    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.066975 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.066952    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnr4\" (UniqueName: \"kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.167922 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.167896    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.168041 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.167943    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.168041 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.168017    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnr4\" (UniqueName: \"kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.168639 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.168612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.170383 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.170350    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.191256 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.191202    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnr4\" (UniqueName: \"kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4\") pod \"maas-api-revoke-sub-452b6d7b-bqjd9\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.249914 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.249890    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:00.555987 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:00.555950    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"]
Nov 02 16:09:00.557084 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:09:00.557052    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e2ab95_4135_4568_a352_ea82e7688f8c.slice/crio-29faeea979f1b7a1fe6779be72186dc10365bedf13553e5079877b89c2b60292 WatchSource:0}: Error finding container 29faeea979f1b7a1fe6779be72186dc10365bedf13553e5079877b89c2b60292: Status 404 returned error can't find the container with id 29faeea979f1b7a1fe6779be72186dc10365bedf13553e5079877b89c2b60292
Nov 02 16:09:01.459616 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:01.459588    5041 generic.go:356] "Generic (PLEG): container finished" podID="73e2ab95-4135-4568-a352-ea82e7688f8c" containerID="acd14d492dcf74fed5ca5c46460a5459fbbaad977de1ea8aedcc0d99b84e5e5f" exitCode=0
Nov 02 16:09:01.459944 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:01.459663    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9" event={"ID":"73e2ab95-4135-4568-a352-ea82e7688f8c","Type":"ContainerDied","Data":"acd14d492dcf74fed5ca5c46460a5459fbbaad977de1ea8aedcc0d99b84e5e5f"}
Nov 02 16:09:01.459944 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:01.459710    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9" event={"ID":"73e2ab95-4135-4568-a352-ea82e7688f8c","Type":"ContainerStarted","Data":"29faeea979f1b7a1fe6779be72186dc10365bedf13553e5079877b89c2b60292"}
Nov 02 16:09:02.682608 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.682573    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:02.795158 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.795050    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca\") pod \"73e2ab95-4135-4568-a352-ea82e7688f8c\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") "
Nov 02 16:09:02.795158 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.795112    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4\" (UniqueName: \"kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4\") pod \"73e2ab95-4135-4568-a352-ea82e7688f8c\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") "
Nov 02 16:09:02.795453 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.795256    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca\") pod \"73e2ab95-4135-4568-a352-ea82e7688f8c\" (UID: \"73e2ab95-4135-4568-a352-ea82e7688f8c\") "
Nov 02 16:09:02.795811 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.795778    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca" pod "73e2ab95-4135-4568-a352-ea82e7688f8c" (UID: "73e2ab95-4135-4568-a352-ea82e7688f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:09:02.798063 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.798030    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca" pod "73e2ab95-4135-4568-a352-ea82e7688f8c" (UID: "73e2ab95-4135-4568-a352-ea82e7688f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:09:02.798184 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.798158    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4" pod "73e2ab95-4135-4568-a352-ea82e7688f8c" (UID: "73e2ab95-4135-4568-a352-ea82e7688f8c"). InnerVolumeSpecName "kube-api-access-vlnr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:09:02.897265 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.897221    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:02.897265 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.897260    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e2ab95-4135-4568-a352-ea82e7688f8c-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:02.897566 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:02.897277    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlnr4\" (UniqueName: \"kubernetes.io/projected/73e2ab95-4135-4568-a352-ea82e7688f8c-kube-api-access-vlnr4\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:03.493387 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:03.491793    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9" event={"ID":"73e2ab95-4135-4568-a352-ea82e7688f8c","Type":"ContainerDied","Data":"29faeea979f1b7a1fe6779be72186dc10365bedf13553e5079877b89c2b60292"}
Nov 02 16:09:03.493387 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:03.491858    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29faeea979f1b7a1fe6779be72186dc10365bedf13553e5079877b89c2b60292"
Nov 02 16:09:03.493387 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:03.491925    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"
Nov 02 16:09:04.483386 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:04.479758    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:09:04.483386 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:04.479835    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:09:06.768569 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:09:06.768526    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:09:11.227417 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:11.227350    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"]
Nov 02 16:09:13.372453 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:13.372416    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"]
Nov 02 16:09:17.192698 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.192660    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"]
Nov 02 16:09:17.193400 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.193348    5041 cpu_manager.go:414] "containerMap: removing container" podUID="73e2ab95-4135-4568-a352-ea82e7688f8c" containerName="revoke-keys"
Nov 02 16:09:17.193400 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.193386    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e2ab95-4135-4568-a352-ea82e7688f8c" containerName="revoke-keys"
Nov 02 16:09:17.193595 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.193524    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e2ab95-4135-4568-a352-ea82e7688f8c" containerName="revoke-keys"
Nov 02 16:09:17.197485 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.197462    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.218422 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.218392    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"]
Nov 02 16:09:17.225851 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.225827    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:09:17.226017 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.225823    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:09:17.226137 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.225978    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:09:17.226210 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.225993    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:09:17.265967 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.265934    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.266148 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.265981    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.266148 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.266077    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cl78\" (UniqueName: \"kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.366735 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.366703    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cl78\" (UniqueName: \"kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.366883 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.366854    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.366944 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.366889    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.367558 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.367525    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.369162 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.369139    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.390075 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.390051    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cl78\" (UniqueName: \"kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78\") pod \"maas-api-revoke-sub-3e31a506-42nls\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.511816 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.511760    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:17.809522 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:17.809496    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"]
Nov 02 16:09:17.811544 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:09:17.811504    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d880db_075f_47e0_be14_90faa1cd6109.slice/crio-a0a1172ddae7d944528c4f54377b392b15224f6288e468de348ee95286f12f81 WatchSource:0}: Error finding container a0a1172ddae7d944528c4f54377b392b15224f6288e468de348ee95286f12f81: Status 404 returned error can't find the container with id a0a1172ddae7d944528c4f54377b392b15224f6288e468de348ee95286f12f81
Nov 02 16:09:18.620295 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:18.620200    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls" event={"ID":"23d880db-075f-47e0-be14-90faa1cd6109","Type":"ContainerStarted","Data":"f81dba533aba2e4b538f17ca0bb37f7bdae1bb65dfacf7c953254120f0dd404b"}
Nov 02 16:09:18.620295 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:18.620275    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls" event={"ID":"23d880db-075f-47e0-be14-90faa1cd6109","Type":"ContainerStarted","Data":"a0a1172ddae7d944528c4f54377b392b15224f6288e468de348ee95286f12f81"}
Nov 02 16:09:18.661252 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:18.660908    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls" podStartSLOduration=1.660890009 podStartE2EDuration="1.660890009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:09:18.658694642 +0000 UTC m=+4410.916764096" watchObservedRunningTime="2026-11-02 16:09:18.660890009 +0000 UTC m=+4410.918959461"
Nov 02 16:09:19.678388 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:19.674808    5041 generic.go:356] "Generic (PLEG): container finished" podID="23d880db-075f-47e0-be14-90faa1cd6109" containerID="f81dba533aba2e4b538f17ca0bb37f7bdae1bb65dfacf7c953254120f0dd404b" exitCode=0
Nov 02 16:09:19.678388 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:19.674904    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls" event={"ID":"23d880db-075f-47e0-be14-90faa1cd6109","Type":"ContainerDied","Data":"f81dba533aba2e4b538f17ca0bb37f7bdae1bb65dfacf7c953254120f0dd404b"}
Nov 02 16:09:20.834173 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.834147    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:20.877744 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.877717    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78\" (UniqueName: \"kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78\") pod \"23d880db-075f-47e0-be14-90faa1cd6109\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") "
Nov 02 16:09:20.877947 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.877884    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca\") pod \"23d880db-075f-47e0-be14-90faa1cd6109\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") "
Nov 02 16:09:20.878021 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.877954    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca\") pod \"23d880db-075f-47e0-be14-90faa1cd6109\" (UID: \"23d880db-075f-47e0-be14-90faa1cd6109\") "
Nov 02 16:09:20.878426 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.878354    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca" pod "23d880db-075f-47e0-be14-90faa1cd6109" (UID: "23d880db-075f-47e0-be14-90faa1cd6109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:09:20.880521 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.880496    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca" pod "23d880db-075f-47e0-be14-90faa1cd6109" (UID: "23d880db-075f-47e0-be14-90faa1cd6109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:09:20.880636 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.880521    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78" pod "23d880db-075f-47e0-be14-90faa1cd6109" (UID: "23d880db-075f-47e0-be14-90faa1cd6109"). InnerVolumeSpecName "kube-api-access-7cl78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:09:20.979419 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.979311    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:20.979419 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.979342    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d880db-075f-47e0-be14-90faa1cd6109-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:20.979419 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:20.979353    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cl78\" (UniqueName: \"kubernetes.io/projected/23d880db-075f-47e0-be14-90faa1cd6109-kube-api-access-7cl78\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:21.687894 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:21.687867    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"
Nov 02 16:09:21.688047 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:21.687907    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls" event={"ID":"23d880db-075f-47e0-be14-90faa1cd6109","Type":"ContainerDied","Data":"a0a1172ddae7d944528c4f54377b392b15224f6288e468de348ee95286f12f81"}
Nov 02 16:09:21.688047 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:21.687975    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a1172ddae7d944528c4f54377b392b15224f6288e468de348ee95286f12f81"
Nov 02 16:09:24.030703 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:24.030671    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"]
Nov 02 16:09:24.041773 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:24.041734    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6vhl4"]
Nov 02 16:09:25.886121 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:25.886087    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30eed75c-39fc-4e1d-96ea-8bc58db718d4" path="/var/lib/kubelet/pods/30eed75c-39fc-4e1d-96ea-8bc58db718d4/volumes"
Nov 02 16:09:26.248417 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.248327    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" podUID="2f6a288b-c51f-4a05-ab3e-6c4396127123" containerName="main" containerID="cri-o://26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef" gracePeriod=45
Nov 02 16:09:26.271602 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.271568    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" podUID="2f6a288b-c51f-4a05-ab3e-6c4396127123" containerName="main" probeResult="failure" output="Get \"https://10.132.1.101:8000/ready\": dial tcp 10.132.1.101:8000: connect: connection refused"
Nov 02 16:09:26.506880 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.506824    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:09:26.530388 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530348    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir\") pod \"2f6a288b-c51f-4a05-ab3e-6c4396127123\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") "
Nov 02 16:09:26.530492 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530436    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm\") pod \"2f6a288b-c51f-4a05-ab3e-6c4396127123\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") "
Nov 02 16:09:26.530550 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530492    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache\") pod \"2f6a288b-c51f-4a05-ab3e-6c4396127123\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") "
Nov 02 16:09:26.530550 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530515    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs\") pod \"2f6a288b-c51f-4a05-ab3e-6c4396127123\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") "
Nov 02 16:09:26.530550 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530532    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home\") pod \"2f6a288b-c51f-4a05-ab3e-6c4396127123\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") "
Nov 02 16:09:26.530686 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530566    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx\" (UniqueName: \"kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx\") pod \"2f6a288b-c51f-4a05-ab3e-6c4396127123\" (UID: \"2f6a288b-c51f-4a05-ab3e-6c4396127123\") "
Nov 02 16:09:26.530686 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530641    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir" pod "2f6a288b-c51f-4a05-ab3e-6c4396127123" (UID: "2f6a288b-c51f-4a05-ab3e-6c4396127123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:26.530806 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache" pod "2f6a288b-c51f-4a05-ab3e-6c4396127123" (UID: "2f6a288b-c51f-4a05-ab3e-6c4396127123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:26.530868 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530834    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home" pod "2f6a288b-c51f-4a05-ab3e-6c4396127123" (UID: "2f6a288b-c51f-4a05-ab3e-6c4396127123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:26.530952 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530934    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:26.530995 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530961    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:26.530995 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.530978    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:26.532773 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.532715    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs" pod "2f6a288b-c51f-4a05-ab3e-6c4396127123" (UID: "2f6a288b-c51f-4a05-ab3e-6c4396127123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:09:26.532773 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.532749    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx" pod "2f6a288b-c51f-4a05-ab3e-6c4396127123" (UID: "2f6a288b-c51f-4a05-ab3e-6c4396127123"). InnerVolumeSpecName "kube-api-access-ggcsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:09:26.532939 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.532801    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm" pod "2f6a288b-c51f-4a05-ab3e-6c4396127123" (UID: "2f6a288b-c51f-4a05-ab3e-6c4396127123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:26.631991 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.631959    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6a288b-c51f-4a05-ab3e-6c4396127123-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:26.631991 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.631990    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6a288b-c51f-4a05-ab3e-6c4396127123-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:26.632159 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.632001    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggcsx\" (UniqueName: \"kubernetes.io/projected/2f6a288b-c51f-4a05-ab3e-6c4396127123-kube-api-access-ggcsx\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:26.717683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.717654    5041 generic.go:356] "Generic (PLEG): container finished" podID="2f6a288b-c51f-4a05-ab3e-6c4396127123" containerID="26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef" exitCode=0
Nov 02 16:09:26.717817 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.717700    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"
Nov 02 16:09:26.717817 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.717742    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" event={"ID":"2f6a288b-c51f-4a05-ab3e-6c4396127123","Type":"ContainerDied","Data":"26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef"}
Nov 02 16:09:26.717817 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.717787    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd" event={"ID":"2f6a288b-c51f-4a05-ab3e-6c4396127123","Type":"ContainerDied","Data":"4582e7de83c0a9428e8a73f222cebc859183c311e08cb487b90c40901ebc9158"}
Nov 02 16:09:26.717817 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.717806    5041 scope.go:122] "RemoveContainer" containerID="26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef"
Nov 02 16:09:26.727805 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.727787    5041 scope.go:122] "RemoveContainer" containerID="26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef"
Nov 02 16:09:26.728051 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:09:26.728031    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef\": container with ID starting with 26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef not found: ID does not exist" containerID="26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef"
Nov 02 16:09:26.728091 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.728060    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef"} err="failed to get container status \"26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef\": rpc error: code = NotFound desc = could not find container \"26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef\": container with ID starting with 26884c28c1dd2a9852c7805a45c76f5159b7bd0c31d4154f34a7d8d6945629ef not found: ID does not exist"
Nov 02 16:09:26.767664 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.767615    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"]
Nov 02 16:09:26.772750 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:26.772725    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1beeeb25-kserve-6f84cdc8b5-qblvd"]
Nov 02 16:09:27.885890 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:27.885857    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6a288b-c51f-4a05-ab3e-6c4396127123" path="/var/lib/kubelet/pods/2f6a288b-c51f-4a05-ab3e-6c4396127123/volumes"
Nov 02 16:09:28.393491 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.393446    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" podUID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" containerName="main" containerID="cri-o://1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22" gracePeriod=45
Nov 02 16:09:28.419677 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.419645    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" podUID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.102:8000/ready\": dial tcp 10.132.1.102:8000: connect: connection refused"
Nov 02 16:09:28.654405 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.654336    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:09:28.731379 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.731336    5041 generic.go:356] "Generic (PLEG): container finished" podID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" containerID="1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22" exitCode=0
Nov 02 16:09:28.731545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.731403    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"
Nov 02 16:09:28.731545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.731406    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" event={"ID":"73ae991f-b722-49fb-8f40-f2c9f2c99fc1","Type":"ContainerDied","Data":"1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22"}
Nov 02 16:09:28.731545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.731510    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl" event={"ID":"73ae991f-b722-49fb-8f40-f2c9f2c99fc1","Type":"ContainerDied","Data":"31a613e0226a160a06fb8266e8e10e42933dd0e745c0a068417b9b5324ff031f"}
Nov 02 16:09:28.731545 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.731531    5041 scope.go:122] "RemoveContainer" containerID="1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22"
Nov 02 16:09:28.746010 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.745990    5041 scope.go:122] "RemoveContainer" containerID="1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22"
Nov 02 16:09:28.746286 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:09:28.746265    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22\": container with ID starting with 1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22 not found: ID does not exist" containerID="1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22"
Nov 02 16:09:28.746332 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.746294    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22"} err="failed to get container status \"1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22\": rpc error: code = NotFound desc = could not find container \"1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22\": container with ID starting with 1ddcb87c2861f97528b6ce25ea421bd008c997b8b84ad8096dc29e309bb68a22 not found: ID does not exist"
Nov 02 16:09:28.747565 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747546    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir\") pod \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") "
Nov 02 16:09:28.747635 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747597    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs\") pod \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") "
Nov 02 16:09:28.747690 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm\") pod \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") "
Nov 02 16:09:28.747787 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x\" (UniqueName: \"kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x\") pod \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") "
Nov 02 16:09:28.747849 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747821    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir" pod "73ae991f-b722-49fb-8f40-f2c9f2c99fc1" (UID: "73ae991f-b722-49fb-8f40-f2c9f2c99fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:28.747907 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747866    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache\") pod \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") "
Nov 02 16:09:28.747907 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.747886    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home\") pod \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\" (UID: \"73ae991f-b722-49fb-8f40-f2c9f2c99fc1\") "
Nov 02 16:09:28.748107 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.748084    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache" pod "73ae991f-b722-49fb-8f40-f2c9f2c99fc1" (UID: "73ae991f-b722-49fb-8f40-f2c9f2c99fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:28.748209 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.748189    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home" pod "73ae991f-b722-49fb-8f40-f2c9f2c99fc1" (UID: "73ae991f-b722-49fb-8f40-f2c9f2c99fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:28.748321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.748307    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-model-cache\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:28.748394 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.748327    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-home\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:28.748394 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.748341    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tmp-dir\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:28.749696 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.749676    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm" pod "73ae991f-b722-49fb-8f40-f2c9f2c99fc1" (UID: "73ae991f-b722-49fb-8f40-f2c9f2c99fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:09:28.749766 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.749751    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs" pod "73ae991f-b722-49fb-8f40-f2c9f2c99fc1" (UID: "73ae991f-b722-49fb-8f40-f2c9f2c99fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:09:28.749859 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.749839    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x" pod "73ae991f-b722-49fb-8f40-f2c9f2c99fc1" (UID: "73ae991f-b722-49fb-8f40-f2c9f2c99fc1"). InnerVolumeSpecName "kube-api-access-hrh5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:09:28.849424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.849394    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrh5x\" (UniqueName: \"kubernetes.io/projected/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-kube-api-access-hrh5x\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:28.849424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.849422    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-tls-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:28.849424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:28.849431    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ae991f-b722-49fb-8f40-f2c9f2c99fc1-dshm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:29.085380 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:29.085338    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"]
Nov 02 16:09:29.089332 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:29.089299    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1beeeb25-kserve-698d575897-qtckl"]
Nov 02 16:09:29.923444 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:29.923406    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" path="/var/lib/kubelet/pods/73ae991f-b722-49fb-8f40-f2c9f2c99fc1/volumes"
Nov 02 16:09:34.494539 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:34.494488    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jx4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:09:34.495390 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:34.495330    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jx4lc" podUID="67aac315-0378-4703-bd0b-67c163800196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:09:36.768177 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:09:36.768134    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:09:40.774438 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774403    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"]
Nov 02 16:09:40.774839 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774797    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2f6a288b-c51f-4a05-ab3e-6c4396127123" containerName="main"
Nov 02 16:09:40.774839 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774809    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6a288b-c51f-4a05-ab3e-6c4396127123" containerName="main"
Nov 02 16:09:40.774839 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774820    5041 cpu_manager.go:414] "containerMap: removing container" podUID="23d880db-075f-47e0-be14-90faa1cd6109" containerName="revoke-keys"
Nov 02 16:09:40.774839 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774826    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d880db-075f-47e0-be14-90faa1cd6109" containerName="revoke-keys"
Nov 02 16:09:40.774982 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774853    5041 cpu_manager.go:414] "containerMap: removing container" podUID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" containerName="main"
Nov 02 16:09:40.774982 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774858    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" containerName="main"
Nov 02 16:09:40.774982 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774926    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6a288b-c51f-4a05-ab3e-6c4396127123" containerName="main"
Nov 02 16:09:40.774982 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774934    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d880db-075f-47e0-be14-90faa1cd6109" containerName="revoke-keys"
Nov 02 16:09:40.774982 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.774945    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ae991f-b722-49fb-8f40-f2c9f2c99fc1" containerName="main"
Nov 02 16:09:40.779500 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.779475    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.805226 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.805194    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:09:40.805406 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.805218    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:09:40.805406 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.805193    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:09:40.805406 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.805326    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:09:40.809294 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.809271    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"]
Nov 02 16:09:40.894631 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.894604    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.894795 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.894654    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stc7j\" (UniqueName: \"kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.894795 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.894675    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.995823 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.995795    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.995956 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.995845    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stc7j\" (UniqueName: \"kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.995956 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.995866    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.996571 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.996547    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:40.998068 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:40.998049    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:41.049422 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:41.049341    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stc7j\" (UniqueName: \"kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j\") pod \"maas-api-revoke-sub-3835578d-gch6l\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:41.092915 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:41.092891    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:41.392286 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:41.392254    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"]
Nov 02 16:09:41.394229 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:09:41.394198    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41aa7dd8_7145_4ba1_9131_aab46a09e547.slice/crio-bb9db4a045ae7050ac572f563ebfc8a1a072d7fccd30968345b7c191c64e1545 WatchSource:0}: Error finding container bb9db4a045ae7050ac572f563ebfc8a1a072d7fccd30968345b7c191c64e1545: Status 404 returned error can't find the container with id bb9db4a045ae7050ac572f563ebfc8a1a072d7fccd30968345b7c191c64e1545
Nov 02 16:09:41.932825 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:41.932784    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l" event={"ID":"41aa7dd8-7145-4ba1-9131-aab46a09e547","Type":"ContainerStarted","Data":"c812935a8fc49d38559565b034f68b3f2fc9e877292737b20f9140f251be3e85"}
Nov 02 16:09:41.933194 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:41.932848    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l" event={"ID":"41aa7dd8-7145-4ba1-9131-aab46a09e547","Type":"ContainerStarted","Data":"bb9db4a045ae7050ac572f563ebfc8a1a072d7fccd30968345b7c191c64e1545"}
Nov 02 16:09:41.973133 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:41.973043    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l" podStartSLOduration=1.973030686 podStartE2EDuration="1.973030686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:09:41.971423254 +0000 UTC m=+4434.229492703" watchObservedRunningTime="2026-11-02 16:09:41.973030686 +0000 UTC m=+4434.231100135"
Nov 02 16:09:42.940710 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:42.940669    5041 generic.go:356] "Generic (PLEG): container finished" podID="41aa7dd8-7145-4ba1-9131-aab46a09e547" containerID="c812935a8fc49d38559565b034f68b3f2fc9e877292737b20f9140f251be3e85" exitCode=0
Nov 02 16:09:42.941213 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:42.940748    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l" event={"ID":"41aa7dd8-7145-4ba1-9131-aab46a09e547","Type":"ContainerDied","Data":"c812935a8fc49d38559565b034f68b3f2fc9e877292737b20f9140f251be3e85"}
Nov 02 16:09:44.086805 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.086781    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:44.229286 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.229208    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca\") pod \"41aa7dd8-7145-4ba1-9131-aab46a09e547\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") "
Nov 02 16:09:44.229286 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.229256    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca\") pod \"41aa7dd8-7145-4ba1-9131-aab46a09e547\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") "
Nov 02 16:09:44.229528 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.229333    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j\" (UniqueName: \"kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j\") pod \"41aa7dd8-7145-4ba1-9131-aab46a09e547\" (UID: \"41aa7dd8-7145-4ba1-9131-aab46a09e547\") "
Nov 02 16:09:44.229675 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.229646    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca" pod "41aa7dd8-7145-4ba1-9131-aab46a09e547" (UID: "41aa7dd8-7145-4ba1-9131-aab46a09e547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:09:44.231472 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.231446    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j" pod "41aa7dd8-7145-4ba1-9131-aab46a09e547" (UID: "41aa7dd8-7145-4ba1-9131-aab46a09e547"). InnerVolumeSpecName "kube-api-access-stc7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:09:44.231472 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.231453    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca" pod "41aa7dd8-7145-4ba1-9131-aab46a09e547" (UID: "41aa7dd8-7145-4ba1-9131-aab46a09e547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:09:44.330377 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.330323    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-xks-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:44.330377 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.330392    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stc7j\" (UniqueName: \"kubernetes.io/projected/41aa7dd8-7145-4ba1-9131-aab46a09e547-kube-api-access-stc7j\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:44.330585 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.330404    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aa7dd8-7145-4ba1-9131-aab46a09e547-maas-api-openshift-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:09:44.953725 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.953686    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"
Nov 02 16:09:44.953910 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.953719    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l" event={"ID":"41aa7dd8-7145-4ba1-9131-aab46a09e547","Type":"ContainerDied","Data":"bb9db4a045ae7050ac572f563ebfc8a1a072d7fccd30968345b7c191c64e1545"}
Nov 02 16:09:44.953910 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:44.953772    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9db4a045ae7050ac572f563ebfc8a1a072d7fccd30968345b7c191c64e1545"
Nov 02 16:09:53.096250 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.096212    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"]
Nov 02 16:09:53.096881 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.096858    5041 cpu_manager.go:414] "containerMap: removing container" podUID="41aa7dd8-7145-4ba1-9131-aab46a09e547" containerName="revoke-keys"
Nov 02 16:09:53.096979 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.096882    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41aa7dd8-7145-4ba1-9131-aab46a09e547" containerName="revoke-keys"
Nov 02 16:09:53.097083 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.097060    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="41aa7dd8-7145-4ba1-9131-aab46a09e547" containerName="revoke-keys"
Nov 02 16:09:53.101671 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.101652    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.117970 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.117945    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"]
Nov 02 16:09:53.121586 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.121568    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.126570 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.126543    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 16:09:53.126725 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.126705    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cz4f2\""
Nov 02 16:09:53.130327 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.130031    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:09:53.130802 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.130777    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:09:53.134741 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.134711    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"]
Nov 02 16:09:53.145320 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.145292    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"]
Nov 02 16:09:53.214058 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214021    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9646\" (UniqueName: \"kubernetes.io/projected/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-kube-api-access-r9646\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.214234 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214068    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6nb\" (UniqueName: \"kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.214234 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214154    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.214234 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214194    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-metrics-certs\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.214234 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214214    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-openshift-service-ca\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.214435 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214314    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.214435 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214384    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-maas-api-tls\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.214435 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.214427    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.217353 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.217333    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 16:09:53.315819 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.315735    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-metrics-certs\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.315819 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.315777    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-openshift-service-ca\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.315819 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.315812    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.316111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.315872    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-maas-api-tls\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.316111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.315930    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.316111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.315998    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9646\" (UniqueName: \"kubernetes.io/projected/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-kube-api-access-r9646\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.316111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.316029    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6nb\" (UniqueName: \"kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.316111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.316076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.316698 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.316667    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-openshift-service-ca\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.316820 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.316702    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.318535 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.318509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.318647 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.318539    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.318891 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.318870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-maas-api-tls\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.318933 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.318870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-metrics-certs\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.338659 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.338632    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6nb\" (UniqueName: \"kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb\") pod \"maas-api-59cd689d68-qb2bc\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.339232 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.339211    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9646\" (UniqueName: \"kubernetes.io/projected/6ae55c91-d282-42b4-b3da-ec5bb45c5b3f-kube-api-access-r9646\") pod \"maas-api-5484769d88-hd5nz\" (UID: \"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.416377 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.416340    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:53.437877 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.437855    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:53.719262 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.719223    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"]
Nov 02 16:09:53.720628 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:09:53.720600    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4ae90b_32e4_4bc8_988e_1dcffbcaef03.slice/crio-4ed2f3b477af8991de23f290a1ac64cd67c7d22f048f4e7bbc1953059a48ede8 WatchSource:0}: Error finding container 4ed2f3b477af8991de23f290a1ac64cd67c7d22f048f4e7bbc1953059a48ede8: Status 404 returned error can't find the container with id 4ed2f3b477af8991de23f290a1ac64cd67c7d22f048f4e7bbc1953059a48ede8
Nov 02 16:09:53.722118 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.722098    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 16:09:53.756094 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:53.756065    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"]
Nov 02 16:09:53.757526 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:09:53.757498    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae55c91_d282_42b4_b3da_ec5bb45c5b3f.slice/crio-cb544263c29f1dbe7f23f19913e2e2f262e60858627d40e1825fad1ac083b693 WatchSource:0}: Error finding container cb544263c29f1dbe7f23f19913e2e2f262e60858627d40e1825fad1ac083b693: Status 404 returned error can't find the container with id cb544263c29f1dbe7f23f19913e2e2f262e60858627d40e1825fad1ac083b693
Nov 02 16:09:54.004682 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:54.004593    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc" event={"ID":"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03","Type":"ContainerStarted","Data":"4ed2f3b477af8991de23f290a1ac64cd67c7d22f048f4e7bbc1953059a48ede8"}
Nov 02 16:09:54.005900 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:54.005862    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz" event={"ID":"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f","Type":"ContainerStarted","Data":"cb544263c29f1dbe7f23f19913e2e2f262e60858627d40e1825fad1ac083b693"}
Nov 02 16:09:55.013230 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:55.013180    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc" event={"ID":"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03","Type":"ContainerStarted","Data":"0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85"}
Nov 02 16:09:55.013758 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:55.013374    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:09:55.014899 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:55.014874    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz" event={"ID":"6ae55c91-d282-42b4-b3da-ec5bb45c5b3f","Type":"ContainerStarted","Data":"a27eb36007d1148e6721cb2831c5fec8fc6017e8fb4f565d48c5dc400923f62d"}
Nov 02 16:09:55.015015 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:55.014923    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:09:55.056307 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:55.056249    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc" podStartSLOduration=1.578334554 podStartE2EDuration="2.056234791s" totalImagesPullingTime="477.900237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:09:55.054802027 +0000 UTC m=+4447.312871477" watchObservedRunningTime="2026-11-02 16:09:55.056234791 +0000 UTC m=+4447.314304281"
Nov 02 16:09:55.098508 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:09:55.098441    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz" podStartSLOduration=1.741009686 podStartE2EDuration="2.098421101s" totalImagesPullingTime="357.411415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:09:55.096428943 +0000 UTC m=+4447.354498394" watchObservedRunningTime="2026-11-02 16:09:55.098421101 +0000 UTC m=+4447.356490553"
Nov 02 16:10:01.033842 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.033643    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:10:01.034261 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.034065    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5484769d88-hd5nz"
Nov 02 16:10:01.152488 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.152454    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"]
Nov 02 16:10:01.152667 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.152626    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc" podUID="8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" containerName="maas-api" containerID="cri-o://0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85" gracePeriod=30
Nov 02 16:10:01.411571 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.411545    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:10:01.498432 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.498400    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls\") pod \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") "
Nov 02 16:10:01.498581 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.498523    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs\") pod \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") "
Nov 02 16:10:01.498581 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.498561    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca\") pod \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") "
Nov 02 16:10:01.498675 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.498629    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb\" (UniqueName: \"kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb\") pod \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\" (UID: \"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03\") "
Nov 02 16:10:01.499019 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.498989    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca" pod "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" (UID: "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:01.500528 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.500505    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls" pod "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" (UID: "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:01.500628 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.500524    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs" pod "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" (UID: "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:01.500628 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.500602    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb" pod "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" (UID: "8c4ae90b-32e4-4bc8-988e-1dcffbcaef03"). InnerVolumeSpecName "kube-api-access-rd6nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:01.599326 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.599259    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:01.599326 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.599288    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:01.599326 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.599298    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:01.599326 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:01.599306    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd6nb\" (UniqueName: \"kubernetes.io/projected/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03-kube-api-access-rd6nb\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:02.071668 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.071617    5041 generic.go:356] "Generic (PLEG): container finished" podID="8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" containerID="0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85" exitCode=0
Nov 02 16:10:02.072074 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.071703    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc" event={"ID":"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03","Type":"ContainerDied","Data":"0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85"}
Nov 02 16:10:02.072074 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.071738    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"
Nov 02 16:10:02.072074 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.071818    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc" event={"ID":"8c4ae90b-32e4-4bc8-988e-1dcffbcaef03","Type":"ContainerDied","Data":"4ed2f3b477af8991de23f290a1ac64cd67c7d22f048f4e7bbc1953059a48ede8"}
Nov 02 16:10:02.072074 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.071847    5041 scope.go:122] "RemoveContainer" containerID="0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85"
Nov 02 16:10:02.081530 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.081500    5041 scope.go:122] "RemoveContainer" containerID="0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85"
Nov 02 16:10:02.081808 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:02.081772    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85\": container with ID starting with 0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85 not found: ID does not exist" containerID="0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85"
Nov 02 16:10:02.081866 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.081831    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85"} err="failed to get container status \"0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85\": rpc error: code = NotFound desc = could not find container \"0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85\": container with ID starting with 0a00167f992aaf4b194e1ce595c13a57785e3972f3f3d508cdfbf1050bf8ce85 not found: ID does not exist"
Nov 02 16:10:02.123755 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.123723    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"]
Nov 02 16:10:02.134883 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:02.134858    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-qb2bc"]
Nov 02 16:10:03.886178 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:03.886145    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" path="/var/lib/kubelet/pods/8c4ae90b-32e4-4bc8-988e-1dcffbcaef03/volumes"
Nov 02 16:10:05.221321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.221286    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"]
Nov 02 16:10:05.221863 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.221760    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" containerName="maas-api"
Nov 02 16:10:05.221863 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.221775    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" containerName="maas-api"
Nov 02 16:10:05.222016 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.221878    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4ae90b-32e4-4bc8-988e-1dcffbcaef03" containerName="maas-api"
Nov 02 16:10:05.226729 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.226695    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.252896 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.252848    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2fa6473d-serving-cert\""
Nov 02 16:10:05.261149 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.258959    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"]
Nov 02 16:10:05.334010 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.333977    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.334221 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.334039    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.334221 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.334144    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.334351 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.334241    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4b46\" (UniqueName: \"kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.412617 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.412574    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"]
Nov 02 16:10:05.416463 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.416441    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.434247 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.434218    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2fa6473d\""
Nov 02 16:10:05.435701 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.435677    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.435852 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.435730    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.435936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.435851    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4b46\" (UniqueName: \"kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.435936 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.435888    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.436449 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:05.436408    5041 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-serving-cert: secret "maas-api-e2e-ait-2fa6473d-serving-cert" not found
Nov 02 16:10:05.436577 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:05.436478    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls podName:dddd0b97-f9f2-4a17-8fb7-887ee4bde478 nodeName:}" failed. No retries permitted until 2026-11-02 16:10:05.936459545 +0000 UTC m=+4458.194528991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls") pod "maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" (UID: "dddd0b97-f9f2-4a17-8fb7-887ee4bde478") : secret "maas-api-e2e-ait-2fa6473d-serving-cert" not found
Nov 02 16:10:05.436577 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.436413    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.438503 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.438478    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.439470 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.439447    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"]
Nov 02 16:10:05.459450 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.459411    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4b46\" (UniqueName: \"kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.480952 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.480880    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"]
Nov 02 16:10:05.484935 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.484914    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.503503 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.503479    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2fa6473d-dockercfg-rhf28\""
Nov 02 16:10:05.506942 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.506917    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"]
Nov 02 16:10:05.537453 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.537403    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.537677 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.537494    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgd9s\" (UniqueName: \"kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s\") pod \"payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") " pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.537677 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.537530    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkzm\" (UniqueName: \"kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm\") pod \"payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.537677 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.537605    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume\") pod \"payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") " pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.638905 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.638859    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.639201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.638918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgd9s\" (UniqueName: \"kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s\") pod \"payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") " pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.639201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.638938    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkzm\" (UniqueName: \"kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm\") pod \"payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.639201 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.638986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume\") pod \"payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") " pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.639711 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.639692    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume\") pod \"payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") " pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.639781 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.639711    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.662304 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.662267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkzm\" (UniqueName: \"kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm\") pod \"payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.662474 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.662267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgd9s\" (UniqueName: \"kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s\") pod \"payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") " pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.730529 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.730495    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:05.799841 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.799789    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:05.945713 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.945671    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:05.948223 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:05.948194    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\") pod \"maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:06.029807 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:06.029779    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"]
Nov 02 16:10:06.031330 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:10:06.031306    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0daf689f_7ca8_4bad_abd9_8a3b9878314c.slice/crio-e81f4de1c65cb2149ef096557a4e34b89e9ab535f308cc8e61db176c40915154 WatchSource:0}: Error finding container e81f4de1c65cb2149ef096557a4e34b89e9ab535f308cc8e61db176c40915154: Status 404 returned error can't find the container with id e81f4de1c65cb2149ef096557a4e34b89e9ab535f308cc8e61db176c40915154
Nov 02 16:10:06.098649 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:06.098614    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8" event={"ID":"0daf689f-7ca8-4bad-abd9-8a3b9878314c","Type":"ContainerStarted","Data":"e81f4de1c65cb2149ef096557a4e34b89e9ab535f308cc8e61db176c40915154"}
Nov 02 16:10:06.109864 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:06.109837    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"]
Nov 02 16:10:06.113085 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:10:06.113057    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55f1ebf_3a1a_42c0_a11e_86a7d07905df.slice/crio-05de68578556f65f1ed557d58d3b7f2ff8f5895aee96c8b09403740d127e4434 WatchSource:0}: Error finding container 05de68578556f65f1ed557d58d3b7f2ff8f5895aee96c8b09403740d127e4434: Status 404 returned error can't find the container with id 05de68578556f65f1ed557d58d3b7f2ff8f5895aee96c8b09403740d127e4434
Nov 02 16:10:06.142665 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:06.142640    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:06.453917 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:06.453866    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"]
Nov 02 16:10:06.769281 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:06.769194    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:10:07.106270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.106182    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" event={"ID":"dddd0b97-f9f2-4a17-8fb7-887ee4bde478","Type":"ContainerStarted","Data":"989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1"}
Nov 02 16:10:07.106270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.106258    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" event={"ID":"dddd0b97-f9f2-4a17-8fb7-887ee4bde478","Type":"ContainerStarted","Data":"c4ea2e3f61c43b08720e7094ddb62619d88b08a2a38478bffd725d6dd9405ef9"}
Nov 02 16:10:07.106523 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.106312    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:07.108146 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.108111    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8" event={"ID":"0daf689f-7ca8-4bad-abd9-8a3b9878314c","Type":"ContainerStarted","Data":"9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9"}
Nov 02 16:10:07.108288 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.108229    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:07.109781 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.109758    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv" event={"ID":"a55f1ebf-3a1a-42c0-a11e-86a7d07905df","Type":"ContainerStarted","Data":"39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0"}
Nov 02 16:10:07.109868 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.109798    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:07.109868 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.109820    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv" event={"ID":"a55f1ebf-3a1a-42c0-a11e-86a7d07905df","Type":"ContainerStarted","Data":"05de68578556f65f1ed557d58d3b7f2ff8f5895aee96c8b09403740d127e4434"}
Nov 02 16:10:07.147022 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.146981    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" podStartSLOduration=1.763789818 podStartE2EDuration="2.146971075s" totalImagesPullingTime="383.181257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:07.146024502 +0000 UTC m=+4459.404093952" watchObservedRunningTime="2026-11-02 16:10:07.146971075 +0000 UTC m=+4459.405040525"
Nov 02 16:10:07.185180 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.185135    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8" podStartSLOduration=2.185121996 podStartE2EDuration="2.185121996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:07.184242335 +0000 UTC m=+4459.442311789" watchObservedRunningTime="2026-11-02 16:10:07.185121996 +0000 UTC m=+4459.443191447"
Nov 02 16:10:07.225916 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:07.225871    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv" podStartSLOduration=2.225856455 podStartE2EDuration="2.225856455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:07.222771163 +0000 UTC m=+4459.480840614" watchObservedRunningTime="2026-11-02 16:10:07.225856455 +0000 UTC m=+4459.483926205"
Nov 02 16:10:13.120745 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:13.120715    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:18.118513 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:18.118475    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:18.118948 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:18.118658    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:29.758124 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:29.758083    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"]
Nov 02 16:10:29.796027 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:29.795988    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"]
Nov 02 16:10:29.897729 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:29.897691    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"]
Nov 02 16:10:29.897960 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:29.897925    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" podUID="dddd0b97-f9f2-4a17-8fb7-887ee4bde478" containerName="maas-api" containerID="cri-o://989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1" gracePeriod=30
Nov 02 16:10:30.157452 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.157423    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:30.194995 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.194965    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\") pod \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") "
Nov 02 16:10:30.195193 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.195031    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs\") pod \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") "
Nov 02 16:10:30.195193 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.195063    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca\") pod \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") "
Nov 02 16:10:30.195193 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.195153    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46\" (UniqueName: \"kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46\") pod \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\" (UID: \"dddd0b97-f9f2-4a17-8fb7-887ee4bde478\") "
Nov 02 16:10:30.195551 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.195518    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca" pod "dddd0b97-f9f2-4a17-8fb7-887ee4bde478" (UID: "dddd0b97-f9f2-4a17-8fb7-887ee4bde478"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:30.197410 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.197380    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs" pod "dddd0b97-f9f2-4a17-8fb7-887ee4bde478" (UID: "dddd0b97-f9f2-4a17-8fb7-887ee4bde478"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:30.197410 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.197393    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls" pod "dddd0b97-f9f2-4a17-8fb7-887ee4bde478" (UID: "dddd0b97-f9f2-4a17-8fb7-887ee4bde478"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:30.197546 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.197406    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46" pod "dddd0b97-f9f2-4a17-8fb7-887ee4bde478" (UID: "dddd0b97-f9f2-4a17-8fb7-887ee4bde478"). InnerVolumeSpecName "kube-api-access-b4b46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:30.248319 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.248284    5041 generic.go:356] "Generic (PLEG): container finished" podID="dddd0b97-f9f2-4a17-8fb7-887ee4bde478" containerID="989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1" exitCode=0
Nov 02 16:10:30.248499 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.248327    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"
Nov 02 16:10:30.248499 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.248378    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" event={"ID":"dddd0b97-f9f2-4a17-8fb7-887ee4bde478","Type":"ContainerDied","Data":"989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1"}
Nov 02 16:10:30.248499 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.248432    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn" event={"ID":"dddd0b97-f9f2-4a17-8fb7-887ee4bde478","Type":"ContainerDied","Data":"c4ea2e3f61c43b08720e7094ddb62619d88b08a2a38478bffd725d6dd9405ef9"}
Nov 02 16:10:30.248499 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.248453    5041 scope.go:122] "RemoveContainer" containerID="989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1"
Nov 02 16:10:30.258215 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.258192    5041 scope.go:122] "RemoveContainer" containerID="989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1"
Nov 02 16:10:30.258519 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:30.258499    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1\": container with ID starting with 989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1 not found: ID does not exist" containerID="989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1"
Nov 02 16:10:30.258601 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.258531    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1"} err="failed to get container status \"989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1\": rpc error: code = NotFound desc = could not find container \"989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1\": container with ID starting with 989aa550ca26a482e2afd8ea632a77ebecde93283bb50ce54e001cb10f12b3b1 not found: ID does not exist"
Nov 02 16:10:30.296625 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.296569    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4b46\" (UniqueName: \"kubernetes.io/projected/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-kube-api-access-b4b46\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:30.296625 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.296593    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:30.296625 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.296603    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:30.296625 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.296611    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddd0b97-f9f2-4a17-8fb7-887ee4bde478-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:30.297983 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.297960    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"]
Nov 02 16:10:30.304243 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:30.304221    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2fa6473d-7ccc9c4d4d-hskgn"]
Nov 02 16:10:31.886421 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:31.886390    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddd0b97-f9f2-4a17-8fb7-887ee4bde478" path="/var/lib/kubelet/pods/dddd0b97-f9f2-4a17-8fb7-887ee4bde478/volumes"
Nov 02 16:10:34.758998 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:34.758947    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8" podUID="0daf689f-7ca8-4bad-abd9-8a3b9878314c" containerName="payload-processing" containerID="cri-o://9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9" gracePeriod=25
Nov 02 16:10:34.797251 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:34.797188    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv" podUID="a55f1ebf-3a1a-42c0-a11e-86a7d07905df" containerName="payload-pre-processing" containerID="cri-o://39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0" gracePeriod=25
Nov 02 16:10:35.048230 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.048198    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:35.087868 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.087842    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:35.141804 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.141770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume\") pod \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") "
Nov 02 16:10:35.141997 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.141828    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume\") pod \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") "
Nov 02 16:10:35.141997 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.141944    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s\" (UniqueName: \"kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s\") pod \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\" (UID: \"0daf689f-7ca8-4bad-abd9-8a3b9878314c\") "
Nov 02 16:10:35.142119 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.142003    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm\" (UniqueName: \"kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm\") pod \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\" (UID: \"a55f1ebf-3a1a-42c0-a11e-86a7d07905df\") "
Nov 02 16:10:35.142295 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.142251    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume" pod "0daf689f-7ca8-4bad-abd9-8a3b9878314c" (UID: "0daf689f-7ca8-4bad-abd9-8a3b9878314c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:35.142354 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.142256    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume" pod "a55f1ebf-3a1a-42c0-a11e-86a7d07905df" (UID: "a55f1ebf-3a1a-42c0-a11e-86a7d07905df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:35.144137 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.144110    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s" pod "0daf689f-7ca8-4bad-abd9-8a3b9878314c" (UID: "0daf689f-7ca8-4bad-abd9-8a3b9878314c"). InnerVolumeSpecName "kube-api-access-kgd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:35.144250 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.144156    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm" pod "a55f1ebf-3a1a-42c0-a11e-86a7d07905df" (UID: "a55f1ebf-3a1a-42c0-a11e-86a7d07905df"). InnerVolumeSpecName "kube-api-access-9hkzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:35.242781 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.242747    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:35.242781 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.242775    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgd9s\" (UniqueName: \"kubernetes.io/projected/0daf689f-7ca8-4bad-abd9-8a3b9878314c-kube-api-access-kgd9s\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:35.242781 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.242785    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkzm\" (UniqueName: \"kubernetes.io/projected/a55f1ebf-3a1a-42c0-a11e-86a7d07905df-kube-api-access-9hkzm\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:35.243042 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.242795    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0daf689f-7ca8-4bad-abd9-8a3b9878314c-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:10:35.280762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.280687    5041 generic.go:356] "Generic (PLEG): container finished" podID="0daf689f-7ca8-4bad-abd9-8a3b9878314c" containerID="9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9" exitCode=0
Nov 02 16:10:35.280762 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.280730    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"
Nov 02 16:10:35.280976 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.280774    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8" event={"ID":"0daf689f-7ca8-4bad-abd9-8a3b9878314c","Type":"ContainerDied","Data":"9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9"}
Nov 02 16:10:35.280976 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.280833    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8" event={"ID":"0daf689f-7ca8-4bad-abd9-8a3b9878314c","Type":"ContainerDied","Data":"e81f4de1c65cb2149ef096557a4e34b89e9ab535f308cc8e61db176c40915154"}
Nov 02 16:10:35.280976 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.280863    5041 scope.go:122] "RemoveContainer" containerID="9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9"
Nov 02 16:10:35.282336 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.282310    5041 generic.go:356] "Generic (PLEG): container finished" podID="a55f1ebf-3a1a-42c0-a11e-86a7d07905df" containerID="39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0" exitCode=0
Nov 02 16:10:35.282487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.282377    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"
Nov 02 16:10:35.282487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.282395    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv" event={"ID":"a55f1ebf-3a1a-42c0-a11e-86a7d07905df","Type":"ContainerDied","Data":"39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0"}
Nov 02 16:10:35.282487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.282443    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv" event={"ID":"a55f1ebf-3a1a-42c0-a11e-86a7d07905df","Type":"ContainerDied","Data":"05de68578556f65f1ed557d58d3b7f2ff8f5895aee96c8b09403740d127e4434"}
Nov 02 16:10:35.292935 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.292910    5041 scope.go:122] "RemoveContainer" containerID="9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9"
Nov 02 16:10:35.293201 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:35.293179    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9\": container with ID starting with 9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9 not found: ID does not exist" containerID="9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9"
Nov 02 16:10:35.293297 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.293206    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9"} err="failed to get container status \"9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9\": rpc error: code = NotFound desc = could not find container \"9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9\": container with ID starting with 9f92a562376e5ff519e400bedb90da285d750cf0d4a5d5044ac0f43abc4b85a9 not found: ID does not exist"
Nov 02 16:10:35.293297 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.293224    5041 scope.go:122] "RemoveContainer" containerID="39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0"
Nov 02 16:10:35.301272 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.301254    5041 scope.go:122] "RemoveContainer" containerID="39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0"
Nov 02 16:10:35.301523 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:35.301503    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0\": container with ID starting with 39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0 not found: ID does not exist" containerID="39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0"
Nov 02 16:10:35.301582 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.301528    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0"} err="failed to get container status \"39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0\": rpc error: code = NotFound desc = could not find container \"39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0\": container with ID starting with 39327d6dac5f64eb3ea9ed9ca17fcfdb0638606a92dca733a9aacb7180d3f7e0 not found: ID does not exist"
Nov 02 16:10:35.331674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.331645    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"]
Nov 02 16:10:35.336516 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.336494    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2fa6473d-5d69f69d57-xwxr8"]
Nov 02 16:10:35.368487 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.368449    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"]
Nov 02 16:10:35.373104 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.373084    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2fa6473d-97c67df8c-x2wxv"]
Nov 02 16:10:35.887691 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.887658    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0daf689f-7ca8-4bad-abd9-8a3b9878314c" path="/var/lib/kubelet/pods/0daf689f-7ca8-4bad-abd9-8a3b9878314c/volumes"
Nov 02 16:10:35.888097 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:35.888081    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55f1ebf-3a1a-42c0-a11e-86a7d07905df" path="/var/lib/kubelet/pods/a55f1ebf-3a1a-42c0-a11e-86a7d07905df/volumes"
Nov 02 16:10:36.767838 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:10:36.767803    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:10:37.181824 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.181792    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"]
Nov 02 16:10:37.182329 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182308    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a55f1ebf-3a1a-42c0-a11e-86a7d07905df" containerName="payload-pre-processing"
Nov 02 16:10:37.182411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182330    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55f1ebf-3a1a-42c0-a11e-86a7d07905df" containerName="payload-pre-processing"
Nov 02 16:10:37.182411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182355    5041 cpu_manager.go:414] "containerMap: removing container" podUID="dddd0b97-f9f2-4a17-8fb7-887ee4bde478" containerName="maas-api"
Nov 02 16:10:37.182411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182376    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddd0b97-f9f2-4a17-8fb7-887ee4bde478" containerName="maas-api"
Nov 02 16:10:37.182411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182398    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0daf689f-7ca8-4bad-abd9-8a3b9878314c" containerName="payload-processing"
Nov 02 16:10:37.182411 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182404    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0daf689f-7ca8-4bad-abd9-8a3b9878314c" containerName="payload-processing"
Nov 02 16:10:37.182683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182483    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddd0b97-f9f2-4a17-8fb7-887ee4bde478" containerName="maas-api"
Nov 02 16:10:37.182683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182494    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0daf689f-7ca8-4bad-abd9-8a3b9878314c" containerName="payload-processing"
Nov 02 16:10:37.182683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.182505    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55f1ebf-3a1a-42c0-a11e-86a7d07905df" containerName="payload-pre-processing"
Nov 02 16:10:37.187681 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.187647    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.219450 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.215613    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-a7b20c-openshift-default-dockercfg-mqpdb\""
Nov 02 16:10:37.230516 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.230486    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"]
Nov 02 16:10:37.261764 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.261736    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.261930 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.261774    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.261930 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.261801    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.261930 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.261907    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.262060 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.261993    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.262060 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.262026    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5t2j\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.262060 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.262052    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.262171 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.262158    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.262215 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.262202    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.363716 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.363681    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.363894 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.363750    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5t2j\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.363894 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.363795    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.363894 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.363861    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.364110 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.363945    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.364110 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.364010    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.364110 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.364060    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.364110 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.364093    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.364342 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.364146    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.364954 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.364889    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.365062 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.364963    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.365250 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.365226    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.365652 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.365627    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.365740 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.365699    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.368400 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.367901    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.379801 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.379775    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.396216 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.396182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5t2j\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.400890 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.400862    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token\") pod \"e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") " pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.536793 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.536714    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:37.851214 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.851180    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"]
Nov 02 16:10:37.852679 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:10:37.852644    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d485675_9d32_4152_8e0f_516d9c8d68f6.slice/crio-51b979f4939fbac4a73c540bb946c523f1882f9cd0f61a2f502e2dcd98bdf079 WatchSource:0}: Error finding container 51b979f4939fbac4a73c540bb946c523f1882f9cd0f61a2f502e2dcd98bdf079: Status 404 returned error can't find the container with id 51b979f4939fbac4a73c540bb946c523f1882f9cd0f61a2f502e2dcd98bdf079
Nov 02 16:10:37.854799 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.854762    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:37.854893 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.854828    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:37.854893 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:37.854858    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:38.304069 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:38.304024    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" event={"ID":"8d485675-9d32-4152-8e0f-516d9c8d68f6","Type":"ContainerStarted","Data":"b418b6968bc1889f4521099c1700e7b0afc47e49b964c30d595107e66593cc9d"}
Nov 02 16:10:38.304482 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:38.304092    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" event={"ID":"8d485675-9d32-4152-8e0f-516d9c8d68f6","Type":"ContainerStarted","Data":"51b979f4939fbac4a73c540bb946c523f1882f9cd0f61a2f502e2dcd98bdf079"}
Nov 02 16:10:38.349417 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:38.349349    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podStartSLOduration=1.34933596 podStartE2EDuration="1.34933596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:38.348123917 +0000 UTC m=+4490.606193366" watchObservedRunningTime="2026-11-02 16:10:38.34933596 +0000 UTC m=+4490.607405411"
Nov 02 16:10:38.537710 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:38.537674    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:38.542683 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:38.542657    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:39.310873 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:39.310836    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:39.311811 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:39.311788    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:42.142909 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.142870    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"]
Nov 02 16:10:42.146801 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.146778    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.166995 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.166973    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-a7b20c-serving-cert\""
Nov 02 16:10:42.182222 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.182194    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"]
Nov 02 16:10:42.209647 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.209615    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfcs\" (UniqueName: \"kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.209771 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.209716    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.209771 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.209753    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.209927 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.209777    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.311135 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.311100    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.311321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.311199    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfcs\" (UniqueName: \"kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.311321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.311242    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.311321 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.311269    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.311916 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.311885    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.313759 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.313736    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.313891 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.313837    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.334457 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.334425    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfcs\" (UniqueName: \"kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs\") pod \"maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.460935 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.460899    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:42.624459 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.624424    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"]
Nov 02 16:10:42.629294 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.629270    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.644455 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.644433    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-a7b20c-dockercfg-jrxgx\""
Nov 02 16:10:42.644543 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.644453    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-a7b20c\""
Nov 02 16:10:42.663586 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.663560    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"]
Nov 02 16:10:42.715263 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.715109    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl54n\" (UniqueName: \"kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n\") pod \"payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.715263 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.715188    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.726573 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.726530    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"]
Nov 02 16:10:42.731340 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.731319    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.740309 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.740276    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"]
Nov 02 16:10:42.780754 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:10:42.780724    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e8d3cd_38f2_420a_977d_a1ddec910212.slice/crio-74324984a0137625fb14db0ac7d91898e2a9eb705c0d8622854116dedecbc444 WatchSource:0}: Error finding container 74324984a0137625fb14db0ac7d91898e2a9eb705c0d8622854116dedecbc444: Status 404 returned error can't find the container with id 74324984a0137625fb14db0ac7d91898e2a9eb705c0d8622854116dedecbc444
Nov 02 16:10:42.781967 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.781939    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"]
Nov 02 16:10:42.816011 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.815980    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.816143 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.816046    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl54n\" (UniqueName: \"kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n\") pod \"payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.816143 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.816122    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.816310 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.816163    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzx7r\" (UniqueName: \"kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r\") pod \"payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.816885 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.816861    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.839808 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.839784    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl54n\" (UniqueName: \"kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n\") pod \"payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:42.916736 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.916704    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.916851 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.916791    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzx7r\" (UniqueName: \"kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r\") pod \"payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.917345 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.917324    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.939641 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.939616    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzx7r\" (UniqueName: \"kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r\") pod \"payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:42.943387 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:42.943338    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:43.050198 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.050148    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:43.241150 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.241118    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"]
Nov 02 16:10:43.338786 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.338755    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm" event={"ID":"f35f7f5c-6f6f-49b4-ba35-6330902a1e49","Type":"ContainerStarted","Data":"0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e"}
Nov 02 16:10:43.338942 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.338814    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm" event={"ID":"f35f7f5c-6f6f-49b4-ba35-6330902a1e49","Type":"ContainerStarted","Data":"93721e4371ff38b5b184834978f16f655056ed353e2440f95fe49f356e14fb11"}
Nov 02 16:10:43.338942 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.338844    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:43.341512 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.341263    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q" event={"ID":"92e8d3cd-38f2-420a-977d-a1ddec910212","Type":"ContainerStarted","Data":"a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c"}
Nov 02 16:10:43.341512 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.341320    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q" event={"ID":"92e8d3cd-38f2-420a-977d-a1ddec910212","Type":"ContainerStarted","Data":"74324984a0137625fb14db0ac7d91898e2a9eb705c0d8622854116dedecbc444"}
Nov 02 16:10:43.341929 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.341905    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:43.368663 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.362352    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"]
Nov 02 16:10:43.368663 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:10:43.366992    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca4ab4e_5957_4a67_9845_133575fa1cf7.slice/crio-b36231a81cb70fee36079a1cd2c6ca64741e14bfc3f41774884d9db51622afbe WatchSource:0}: Error finding container b36231a81cb70fee36079a1cd2c6ca64741e14bfc3f41774884d9db51622afbe: Status 404 returned error can't find the container with id b36231a81cb70fee36079a1cd2c6ca64741e14bfc3f41774884d9db51622afbe
Nov 02 16:10:43.384144 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.384086    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm" podStartSLOduration=1.3840700799999999 podStartE2EDuration="1.38407008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:43.37969583 +0000 UTC m=+4495.637765277" watchObservedRunningTime="2026-11-02 16:10:43.38407008 +0000 UTC m=+4495.642139529"
Nov 02 16:10:43.422222 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:43.422175    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q" podStartSLOduration=1.074542157 podStartE2EDuration="1.422163033s" totalImagesPullingTime="347.620876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:43.420054604 +0000 UTC m=+4495.678124054" watchObservedRunningTime="2026-11-02 16:10:43.422163033 +0000 UTC m=+4495.680232480"
Nov 02 16:10:44.348769 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:44.348732    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b" event={"ID":"1ca4ab4e-5957-4a67-9845-133575fa1cf7","Type":"ContainerStarted","Data":"91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb"}
Nov 02 16:10:44.349234 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:44.348788    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b" event={"ID":"1ca4ab4e-5957-4a67-9845-133575fa1cf7","Type":"ContainerStarted","Data":"b36231a81cb70fee36079a1cd2c6ca64741e14bfc3f41774884d9db51622afbe"}
Nov 02 16:10:44.349234 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:44.348957    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:44.389521 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:44.389474    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b" podStartSLOduration=2.389461846 podStartE2EDuration="2.389461846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:44.389142771 +0000 UTC m=+4496.647212220" watchObservedRunningTime="2026-11-02 16:10:44.389461846 +0000 UTC m=+4496.647531295"
Nov 02 16:10:49.483243 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:49.483212    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:10:49.510595 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:49.510574    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:10:50.360686 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:50.360654    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:10:53.020280 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:53.020246    5041 scope.go:122] "RemoveContainer" containerID="a65719a1d420c062997bf8030d47c6deb68fd50cd665ab5fd6cb6998956411a0"
Nov 02 16:10:54.352332 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:54.352298    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:10:55.352575 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:55.352524    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 16:10:55.353006 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:55.352608    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 16:10:55.359419 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:55.359394    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:10:56.424270 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:56.424234    5041 generic.go:356] "Generic (PLEG): container finished" podID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerID="b418b6968bc1889f4521099c1700e7b0afc47e49b964c30d595107e66593cc9d" exitCode=0
Nov 02 16:10:56.424746 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:56.424301    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" event={"ID":"8d485675-9d32-4152-8e0f-516d9c8d68f6","Type":"ContainerDied","Data":"b418b6968bc1889f4521099c1700e7b0afc47e49b964c30d595107e66593cc9d"}
Nov 02 16:10:56.424746 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:56.424649    5041 scope.go:122] "RemoveContainer" containerID="b418b6968bc1889f4521099c1700e7b0afc47e49b964c30d595107e66593cc9d"
Nov 02 16:10:56.425954 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:56.425923    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:56.426048 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:56.426008    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:56.426048 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:56.426042    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:57.444256 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:57.444220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" event={"ID":"8d485675-9d32-4152-8e0f-516d9c8d68f6","Type":"ContainerStarted","Data":"7ce7ea63d2c33bd3b9e1e1172c767823f09e4361507576c0e99f88693f5044fe"}
Nov 02 16:10:57.538602 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:57.538558    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:10:57.538776 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:57.538699    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 16:10:57.538776 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:57.538755    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 16:10:58.537956 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:58.537919    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 16:10:58.538511 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:58.538455    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 16:10:59.537777 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:59.537736    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 16:10:59.537955 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:10:59.537797    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 16:11:00.537890 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:00.537845    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 16:11:00.538485 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:00.537914    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 16:11:01.537725 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:01.537687    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 16:11:01.538023 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:01.537743    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 16:11:02.547758 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:02.547718    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:11:02.548255 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:02.548228    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:11:02.549202 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:02.549171    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:11:06.768352 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:11:06.768316    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:11:17.028076 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:17.028037    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"]
Nov 02 16:11:17.039108 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:17.039072    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bmmms"]
Nov 02 16:11:17.887203 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:17.887170    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7913fcda-8e62-4781-9188-982f315f8e4d" path="/var/lib/kubelet/pods/7913fcda-8e62-4781-9188-982f315f8e4d/volumes"
Nov 02 16:11:36.767539 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:11:36.767456    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:11:38.029806 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:38.029762    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"]
Nov 02 16:11:38.040753 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:38.040721    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qjxnw"]
Nov 02 16:11:39.886356 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:11:39.886323    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936e1fcd-88e8-49ed-b5dd-17f769030411" path="/var/lib/kubelet/pods/936e1fcd-88e8-49ed-b5dd-17f769030411/volumes"
Nov 02 16:12:06.768551 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:12:06.768509    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:12:28.925161 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:28.925127    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"]
Nov 02 16:12:28.955797 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:28.955761    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"]
Nov 02 16:12:33.925613 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:33.925559    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b" podUID="1ca4ab4e-5957-4a67-9845-133575fa1cf7" containerName="payload-processing" containerID="cri-o://91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb" gracePeriod=25
Nov 02 16:12:33.956554 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:33.956523    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm" podUID="f35f7f5c-6f6f-49b4-ba35-6330902a1e49" containerName="payload-pre-processing" containerID="cri-o://0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e" gracePeriod=25
Nov 02 16:12:34.211963 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.211294    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:12:34.251710 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.251686    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:12:34.324039 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.324009    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r\" (UniqueName: \"kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r\") pod \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") "
Nov 02 16:12:34.324184 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.324117    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume\") pod \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\" (UID: \"1ca4ab4e-5957-4a67-9845-133575fa1cf7\") "
Nov 02 16:12:34.324564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.324535    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume" pod "1ca4ab4e-5957-4a67-9845-133575fa1cf7" (UID: "1ca4ab4e-5957-4a67-9845-133575fa1cf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:12:34.326163 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.326135    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r" pod "1ca4ab4e-5957-4a67-9845-133575fa1cf7" (UID: "1ca4ab4e-5957-4a67-9845-133575fa1cf7"). InnerVolumeSpecName "kube-api-access-kzx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:12:34.424874 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.424852    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n\" (UniqueName: \"kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n\") pod \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") "
Nov 02 16:12:34.424969 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.424909    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume\") pod \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\" (UID: \"f35f7f5c-6f6f-49b4-ba35-6330902a1e49\") "
Nov 02 16:12:34.425100 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.425088    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzx7r\" (UniqueName: \"kubernetes.io/projected/1ca4ab4e-5957-4a67-9845-133575fa1cf7-kube-api-access-kzx7r\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:34.425141 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.425102    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca4ab4e-5957-4a67-9845-133575fa1cf7-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:34.425284 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.425254    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume" pod "f35f7f5c-6f6f-49b4-ba35-6330902a1e49" (UID: "f35f7f5c-6f6f-49b4-ba35-6330902a1e49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:12:34.426781 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.426761    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n" pod "f35f7f5c-6f6f-49b4-ba35-6330902a1e49" (UID: "f35f7f5c-6f6f-49b4-ba35-6330902a1e49"). InnerVolumeSpecName "kube-api-access-vl54n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:12:34.525520 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.525465    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl54n\" (UniqueName: \"kubernetes.io/projected/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-kube-api-access-vl54n\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:34.525520 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:34.525488    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35f7f5c-6f6f-49b4-ba35-6330902a1e49-plugins-config-volume\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:35.012564 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.012536    5041 generic.go:356] "Generic (PLEG): container finished" podID="1ca4ab4e-5957-4a67-9845-133575fa1cf7" containerID="91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb" exitCode=0
Nov 02 16:12:35.013007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.012603    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"
Nov 02 16:12:35.013007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.012623    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b" event={"ID":"1ca4ab4e-5957-4a67-9845-133575fa1cf7","Type":"ContainerDied","Data":"91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb"}
Nov 02 16:12:35.013007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.012676    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b" event={"ID":"1ca4ab4e-5957-4a67-9845-133575fa1cf7","Type":"ContainerDied","Data":"b36231a81cb70fee36079a1cd2c6ca64741e14bfc3f41774884d9db51622afbe"}
Nov 02 16:12:35.013007 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.012706    5041 scope.go:122] "RemoveContainer" containerID="91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb"
Nov 02 16:12:35.014166 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.014146    5041 generic.go:356] "Generic (PLEG): container finished" podID="f35f7f5c-6f6f-49b4-ba35-6330902a1e49" containerID="0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e" exitCode=0
Nov 02 16:12:35.014268 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.014189    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"
Nov 02 16:12:35.014268 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.014219    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm" event={"ID":"f35f7f5c-6f6f-49b4-ba35-6330902a1e49","Type":"ContainerDied","Data":"0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e"}
Nov 02 16:12:35.014268 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.014259    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm" event={"ID":"f35f7f5c-6f6f-49b4-ba35-6330902a1e49","Type":"ContainerDied","Data":"93721e4371ff38b5b184834978f16f655056ed353e2440f95fe49f356e14fb11"}
Nov 02 16:12:35.027769 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.027745    5041 scope.go:122] "RemoveContainer" containerID="91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb"
Nov 02 16:12:35.028151 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:12:35.028116    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb\": container with ID starting with 91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb not found: ID does not exist" containerID="91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb"
Nov 02 16:12:35.028271 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.028157    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb"} err="failed to get container status \"91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb\": rpc error: code = NotFound desc = could not find container \"91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb\": container with ID starting with 91215bebe6399f9582acf48bb6a69e259fb7b68dee87814af0501ef16f01c3bb not found: ID does not exist"
Nov 02 16:12:35.028271 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.028190    5041 scope.go:122] "RemoveContainer" containerID="0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e"
Nov 02 16:12:35.037279 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.037260    5041 scope.go:122] "RemoveContainer" containerID="0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e"
Nov 02 16:12:35.037598 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:12:35.037579    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e\": container with ID starting with 0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e not found: ID does not exist" containerID="0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e"
Nov 02 16:12:35.037674 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.037604    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e"} err="failed to get container status \"0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e\": rpc error: code = NotFound desc = could not find container \"0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e\": container with ID starting with 0af8d398fbe2166684865b5f70444917f889c1f770d951b32c4ee7c71a66a26e not found: ID does not exist"
Nov 02 16:12:35.077967 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.077945    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"]
Nov 02 16:12:35.082958 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.082933    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-a7b20c-775f6bbd5c-pp57b"]
Nov 02 16:12:35.114040 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.114016    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"]
Nov 02 16:12:35.118028 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.118006    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a7b20c-6557d44c47-4d8qm"]
Nov 02 16:12:35.886707 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.886672    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca4ab4e-5957-4a67-9845-133575fa1cf7" path="/var/lib/kubelet/pods/1ca4ab4e-5957-4a67-9845-133575fa1cf7/volumes"
Nov 02 16:12:35.887056 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:35.887042    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35f7f5c-6f6f-49b4-ba35-6330902a1e49" path="/var/lib/kubelet/pods/f35f7f5c-6f6f-49b4-ba35-6330902a1e49/volumes"
Nov 02 16:12:36.768021 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:12:36.767984    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:12:36.982132 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:36.982098    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"]
Nov 02 16:12:36.982349 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:36.982319    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q" podUID="92e8d3cd-38f2-420a-977d-a1ddec910212" containerName="maas-api" containerID="cri-o://a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c" gracePeriod=30
Nov 02 16:12:37.260196 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.260163    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:12:37.452883 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.452852    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs\" (UniqueName: \"kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs\") pod \"92e8d3cd-38f2-420a-977d-a1ddec910212\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") "
Nov 02 16:12:37.453068 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.452893    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs\") pod \"92e8d3cd-38f2-420a-977d-a1ddec910212\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") "
Nov 02 16:12:37.453068 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.452920    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls\") pod \"92e8d3cd-38f2-420a-977d-a1ddec910212\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") "
Nov 02 16:12:37.453068 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.452977    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca\") pod \"92e8d3cd-38f2-420a-977d-a1ddec910212\" (UID: \"92e8d3cd-38f2-420a-977d-a1ddec910212\") "
Nov 02 16:12:37.453474 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.453447    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca" pod "92e8d3cd-38f2-420a-977d-a1ddec910212" (UID: "92e8d3cd-38f2-420a-977d-a1ddec910212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:12:37.455123 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.455096    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs" pod "92e8d3cd-38f2-420a-977d-a1ddec910212" (UID: "92e8d3cd-38f2-420a-977d-a1ddec910212"). InnerVolumeSpecName "kube-api-access-pqfcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:12:37.455219 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.455135    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs" pod "92e8d3cd-38f2-420a-977d-a1ddec910212" (UID: "92e8d3cd-38f2-420a-977d-a1ddec910212"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:12:37.455219 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.455167    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls" pod "92e8d3cd-38f2-420a-977d-a1ddec910212" (UID: "92e8d3cd-38f2-420a-977d-a1ddec910212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:12:37.554452 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.554413    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqfcs\" (UniqueName: \"kubernetes.io/projected/92e8d3cd-38f2-420a-977d-a1ddec910212-kube-api-access-pqfcs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:37.554452 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.554444    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-metrics-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:37.554452 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.554454    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e8d3cd-38f2-420a-977d-a1ddec910212-maas-api-tls\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:37.554700 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:37.554463    5041 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e8d3cd-38f2-420a-977d-a1ddec910212-openshift-service-ca\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:38.035865 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.035832    5041 generic.go:356] "Generic (PLEG): container finished" podID="92e8d3cd-38f2-420a-977d-a1ddec910212" containerID="a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c" exitCode=0
Nov 02 16:12:38.036424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.035875    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"
Nov 02 16:12:38.036424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.035907    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q" event={"ID":"92e8d3cd-38f2-420a-977d-a1ddec910212","Type":"ContainerDied","Data":"a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c"}
Nov 02 16:12:38.036424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.035965    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q" event={"ID":"92e8d3cd-38f2-420a-977d-a1ddec910212","Type":"ContainerDied","Data":"74324984a0137625fb14db0ac7d91898e2a9eb705c0d8622854116dedecbc444"}
Nov 02 16:12:38.036424 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.035987    5041 scope.go:122] "RemoveContainer" containerID="a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c"
Nov 02 16:12:38.045490 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.045472    5041 scope.go:122] "RemoveContainer" containerID="a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c"
Nov 02 16:12:38.045759 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:12:38.045742    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c\": container with ID starting with a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c not found: ID does not exist" containerID="a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c"
Nov 02 16:12:38.045833 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.045763    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c"} err="failed to get container status \"a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c\": rpc error: code = NotFound desc = could not find container \"a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c\": container with ID starting with a7309a94fb553cc9793aef67e3fb9e8cc692d83a67685e4480222b94e272f75c not found: ID does not exist"
Nov 02 16:12:38.085811 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.085788    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"]
Nov 02 16:12:38.097118 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:38.097097    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a7b20c-788bffd54f-bfx4q"]
Nov 02 16:12:39.886878 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:39.886841    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e8d3cd-38f2-420a-977d-a1ddec910212" path="/var/lib/kubelet/pods/92e8d3cd-38f2-420a-977d-a1ddec910212/volumes"
Nov 02 16:12:44.791569 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:44.791528    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"]
Nov 02 16:12:44.792034 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:44.791807    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" containerID="cri-o://7ce7ea63d2c33bd3b9e1e1172c767823f09e4361507576c0e99f88693f5044fe" gracePeriod=30
Nov 02 16:12:47.550333 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:47.550286    5041 patch_prober.go:29] interesting pod/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 16:12:47.550760 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:47.550385    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 16:12:50.162049 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.161953    5041 generic.go:356] "Generic (PLEG): container finished" podID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerID="7ce7ea63d2c33bd3b9e1e1172c767823f09e4361507576c0e99f88693f5044fe" exitCode=0
Nov 02 16:12:50.166651 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.166615    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" event={"ID":"8d485675-9d32-4152-8e0f-516d9c8d68f6","Type":"ContainerDied","Data":"7ce7ea63d2c33bd3b9e1e1172c767823f09e4361507576c0e99f88693f5044fe"}
Nov 02 16:12:50.166888 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.166869    5041 scope.go:122] "RemoveContainer" containerID="b418b6968bc1889f4521099c1700e7b0afc47e49b964c30d595107e66593cc9d"
Nov 02 16:12:50.548918 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.548879    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:12:50.612585 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612546    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.612757 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612608    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.612757 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612642    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.612757 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612670    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.612757 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612701    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.612757 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612734    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.613051 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612807    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.613051 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612842    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.613051 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.612882    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo\") pod \"8d485675-9d32-4152-8e0f-516d9c8d68f6\" (UID: \"8d485675-9d32-4152-8e0f-516d9c8d68f6\") "
Nov 02 16:12:50.613236 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.613191    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:12:50.613306 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.613257    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:12:50.613395 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.613339    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.613473 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.613427    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:12:50.613539 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.613493    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:12:50.613669 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.613634    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:12:50.615279 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.615251    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "kube-api-access-t5t2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:12:50.615576 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.615551    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:12:50.616847 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.616819    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:12:50.617042 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.617021    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo" pod "8d485675-9d32-4152-8e0f-516d9c8d68f6" (UID: "8d485675-9d32-4152-8e0f-516d9c8d68f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 16:12:50.713827 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713773    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-envoy\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.713827 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713823    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5t2j\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-kube-api-access-t5t2j\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.713827 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713835    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-token\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.714088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713850    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-data\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.714088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713864    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-workload-certs\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.714088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713875    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d485675-9d32-4152-8e0f-516d9c8d68f6-istiod-ca-cert\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.714088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713883    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d485675-9d32-4152-8e0f-516d9c8d68f6-credential-socket\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:50.714088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:50.713894    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d485675-9d32-4152-8e0f-516d9c8d68f6-istio-podinfo\") on node \"ip-10-0-133-143.ec2.internal\" DevicePath \"\""
Nov 02 16:12:51.174105 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:51.174073    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"
Nov 02 16:12:51.174551 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:51.174073    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982" event={"ID":"8d485675-9d32-4152-8e0f-516d9c8d68f6","Type":"ContainerDied","Data":"51b979f4939fbac4a73c540bb946c523f1882f9cd0f61a2f502e2dcd98bdf079"}
Nov 02 16:12:51.174551 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:51.174209    5041 scope.go:122] "RemoveContainer" containerID="7ce7ea63d2c33bd3b9e1e1172c767823f09e4361507576c0e99f88693f5044fe"
Nov 02 16:12:51.237964 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:51.237929    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"]
Nov 02 16:12:51.248757 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:51.248728    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a7b20c-openshift-default-c7dcd4457-wf982"]
Nov 02 16:12:51.886605 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:51.886570    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" path="/var/lib/kubelet/pods/8d485675-9d32-4152-8e0f-516d9c8d68f6/volumes"
Nov 02 16:12:53.078595 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:53.078560    5041 scope.go:122] "RemoveContainer" containerID="6a2e426aab36a9700b8d833581e97115d60a81777e47a342af106c16963662a6"
Nov 02 16:12:53.087962 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:12:53.087939    5041 scope.go:122] "RemoveContainer" containerID="4bcde0e2c9b24bacc3afe2c2e4a2760f8691e74d233e5f2798f6218c83182888"
Nov 02 16:13:06.768434 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:13:06.768329    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:13:36.767679 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:13:36.767634    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:14:03.028532 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:03.028496    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"]
Nov 02 16:14:03.039524 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:03.039495    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bqjd9"]
Nov 02 16:14:03.886112 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:03.886074    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e2ab95-4135-4568-a352-ea82e7688f8c" path="/var/lib/kubelet/pods/73e2ab95-4135-4568-a352-ea82e7688f8c/volumes"
Nov 02 16:14:06.768144 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:14:06.768106    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:14:21.036135 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:21.036100    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"]
Nov 02 16:14:21.047271 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:21.047243    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-42nls"]
Nov 02 16:14:21.886134 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:21.886096    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d880db-075f-47e0-be14-90faa1cd6109" path="/var/lib/kubelet/pods/23d880db-075f-47e0-be14-90faa1cd6109/volumes"
Nov 02 16:14:36.768552 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:14:36.768472    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:14:39.858581 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:39.858553    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wbgvb_717f7a98-82e5-4412-a803-dd3b77fe1c4a/manager/0.log"
Nov 02 16:14:40.363223 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:40.363194    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6667f79cdf-hj9ms_364c4ded-4de4-4cb2-8c46-d9f2e895dcf2/manager/0.log"
Nov 02 16:14:42.028578 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:42.028546    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5484769d88-hd5nz_6ae55c91-d282-42b4-b3da-ec5bb45c5b3f/maas-api/0.log"
Nov 02 16:14:42.417949 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:42.417875    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3835578d-gch6l_41aa7dd8-7145-4ba1-9131-aab46a09e547/revoke-keys/0.log"
Nov 02 16:14:45.028198 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:45.028162    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"]
Nov 02 16:14:45.039099 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:45.039069    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3835578d-gch6l"]
Nov 02 16:14:45.887342 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:45.887308    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41aa7dd8-7145-4ba1-9131-aab46a09e547" path="/var/lib/kubelet/pods/41aa7dd8-7145-4ba1-9131-aab46a09e547/volumes"
Nov 02 16:14:55.068105 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:55.068050    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jxlbw_ac7862b5-e9cf-48af-8cfa-df6763f0e42e/manager/0.log"
Nov 02 16:14:57.419978 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:57.419945    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f469495f-npwt9_9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf/kube-auth-proxy/0.log"
Nov 02 16:14:58.460371 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:58.460344    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-jx4lc_67aac315-0378-4703-bd0b-67c163800196/istio-proxy/0.log"
Nov 02 16:14:58.668111 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:58.668073    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k4x2d_23e8c095-a07b-4484-9fdf-0c87e5392cb8/istio-proxy/0.log"
Nov 02 16:14:58.864853 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:58.864789    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-khzpx_97ddb9f7-6726-49f3-9acc-c5602bcf68d1/istio-proxy/0.log"
Nov 02 16:14:59.063245 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:59.063214    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-rdl5t_03da44d6-bcfd-4e9f-88d7-a9b27157d56b/istio-proxy/0.log"
Nov 02 16:14:59.890289 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:14:59.890258    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-s4wx8_44bb49df-a7f6-426b-812b-df7db3361522/payload-pre-processing/0.log"
Nov 02 16:15:00.281133 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:00.281103    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659675459f-cp5t4_e6beba97-264e-4107-bafa-57b5380ad218/router/0.log"
Nov 02 16:15:01.520837 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:01.520799    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kbk89_39e554bb-b67f-471a-8990-5fa031445730/main/0.log"
Nov 02 16:15:01.918651 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:01.918624    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg_214f8315-f258-40d4-bb11-bfae94d43ff1/main/0.log"
Nov 02 16:15:06.767970 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:15:06.767933    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:15:14.596633 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:14.596600    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wbgvb_717f7a98-82e5-4412-a803-dd3b77fe1c4a/manager/0.log"
Nov 02 16:15:15.043286 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:15.043246    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6667f79cdf-hj9ms_364c4ded-4de4-4cb2-8c46-d9f2e895dcf2/manager/0.log"
Nov 02 16:15:16.461100 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:16.461063    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5484769d88-hd5nz_6ae55c91-d282-42b4-b3da-ec5bb45c5b3f/maas-api/0.log"
Nov 02 16:15:19.061895 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:19.061857    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jxlbw_ac7862b5-e9cf-48af-8cfa-df6763f0e42e/manager/0.log"
Nov 02 16:15:21.007231 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:21.007157    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f469495f-npwt9_9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf/kube-auth-proxy/0.log"
Nov 02 16:15:22.051812 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:22.051776    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-jx4lc_67aac315-0378-4703-bd0b-67c163800196/istio-proxy/0.log"
Nov 02 16:15:22.264887 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:22.264854    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k4x2d_23e8c095-a07b-4484-9fdf-0c87e5392cb8/istio-proxy/0.log"
Nov 02 16:15:22.450412 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:22.450381    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-khzpx_97ddb9f7-6726-49f3-9acc-c5602bcf68d1/istio-proxy/0.log"
Nov 02 16:15:22.658858 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:22.658834    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-rdl5t_03da44d6-bcfd-4e9f-88d7-a9b27157d56b/istio-proxy/0.log"
Nov 02 16:15:23.426106 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:23.426070    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-s4wx8_44bb49df-a7f6-426b-812b-df7db3361522/payload-pre-processing/0.log"
Nov 02 16:15:23.821203 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:23.821115    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659675459f-cp5t4_e6beba97-264e-4107-bafa-57b5380ad218/router/0.log"
Nov 02 16:15:24.774034 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:24.774002    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kbk89_39e554bb-b67f-471a-8990-5fa031445730/main/0.log"
Nov 02 16:15:25.154174 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:25.154105    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jtkvg_214f8315-f258-40d4-bb11-bfae94d43ff1/main/0.log"
Nov 02 16:15:36.767659 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:15:36.767619    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:15:49.521209 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:49.521176    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:15:49.578200 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:49.578177    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlsqw_113ca502-9379-4a88-b3e7-3599fe9cbfd9/console-operator/2.log"
Nov 02 16:15:53.150673 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:53.150640    5041 scope.go:122] "RemoveContainer" containerID="acd14d492dcf74fed5ca5c46460a5459fbbaad977de1ea8aedcc0d99b84e5e5f"
Nov 02 16:15:53.159688 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:53.159668    5041 scope.go:122] "RemoveContainer" containerID="f81dba533aba2e4b538f17ca0bb37f7bdae1bb65dfacf7c953254120f0dd404b"
Nov 02 16:15:53.167598 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:15:53.167572    5041 scope.go:122] "RemoveContainer" containerID="c812935a8fc49d38559565b034f68b3f2fc9e877292737b20f9140f251be3e85"
Nov 02 16:16:06.768171 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:16:06.768088    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:16:10.291627 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:16:10.291591    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6667f79cdf-hj9ms_364c4ded-4de4-4cb2-8c46-d9f2e895dcf2/manager/0.log"
Nov 02 16:16:10.582129 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:16:10.582041    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6667f79cdf-hj9ms_364c4ded-4de4-4cb2-8c46-d9f2e895dcf2/manager/0.log"
Nov 02 16:16:36.767240 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:16:36.767203    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:17:06.769236 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:17:06.769184    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:17:25.485274 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:25.485240    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6667f79cdf-hj9ms_364c4ded-4de4-4cb2-8c46-d9f2e895dcf2/manager/0.log"
Nov 02 16:17:25.773088 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:25.773010    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6667f79cdf-hj9ms_364c4ded-4de4-4cb2-8c46-d9f2e895dcf2/manager/0.log"
Nov 02 16:17:27.022817 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.022770    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xl725/must-gather-4jp9z"]
Nov 02 16:17:27.023349 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023331    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1ca4ab4e-5957-4a67-9845-133575fa1cf7" containerName="payload-processing"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023353    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca4ab4e-5957-4a67-9845-133575fa1cf7" containerName="payload-processing"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023383    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023389    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023397    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023402    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023415    5041 cpu_manager.go:414] "containerMap: removing container" podUID="92e8d3cd-38f2-420a-977d-a1ddec910212" containerName="maas-api"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023420    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e8d3cd-38f2-420a-977d-a1ddec910212" containerName="maas-api"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023439    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f35f7f5c-6f6f-49b4-ba35-6330902a1e49" containerName="payload-pre-processing"
Nov 02 16:17:27.023443 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023443    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35f7f5c-6f6f-49b4-ba35-6330902a1e49" containerName="payload-pre-processing"
Nov 02 16:17:27.023742 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023525    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy"
Nov 02 16:17:27.023742 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023532    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d485675-9d32-4152-8e0f-516d9c8d68f6" containerName="istio-proxy"
Nov 02 16:17:27.023742 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023540    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e8d3cd-38f2-420a-977d-a1ddec910212" containerName="maas-api"
Nov 02 16:17:27.023742 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023549    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35f7f5c-6f6f-49b4-ba35-6330902a1e49" containerName="payload-pre-processing"
Nov 02 16:17:27.023742 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.023559    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca4ab4e-5957-4a67-9845-133575fa1cf7" containerName="payload-processing"
Nov 02 16:17:27.026882 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.026863    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.049035 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.049013    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xl725\"/\"openshift-service-ca.crt\""
Nov 02 16:17:27.049138 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.049046    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xl725\"/\"default-dockercfg-vkz4g\""
Nov 02 16:17:27.049262 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.049249    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xl725\"/\"kube-root-ca.crt\""
Nov 02 16:17:27.065235 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.065208    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/must-gather-4jp9z"]
Nov 02 16:17:27.148401 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.148356    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5kt\" (UniqueName: \"kubernetes.io/projected/41e2ea12-b95d-47b4-bccd-d1a1fc88bd90-kube-api-access-zf5kt\") pod \"must-gather-4jp9z\" (UID: \"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90\") " pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.148554 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.148468    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41e2ea12-b95d-47b4-bccd-d1a1fc88bd90-must-gather-output\") pod \"must-gather-4jp9z\" (UID: \"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90\") " pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.249155 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.249130    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5kt\" (UniqueName: \"kubernetes.io/projected/41e2ea12-b95d-47b4-bccd-d1a1fc88bd90-kube-api-access-zf5kt\") pod \"must-gather-4jp9z\" (UID: \"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90\") " pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.249285 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.249208    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41e2ea12-b95d-47b4-bccd-d1a1fc88bd90-must-gather-output\") pod \"must-gather-4jp9z\" (UID: \"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90\") " pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.249584 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.249566    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41e2ea12-b95d-47b4-bccd-d1a1fc88bd90-must-gather-output\") pod \"must-gather-4jp9z\" (UID: \"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90\") " pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.303041 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.302989    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5kt\" (UniqueName: \"kubernetes.io/projected/41e2ea12-b95d-47b4-bccd-d1a1fc88bd90-kube-api-access-zf5kt\") pod \"must-gather-4jp9z\" (UID: \"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90\") " pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.341617 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.341592    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/must-gather-4jp9z"
Nov 02 16:17:27.644350 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.641333    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/must-gather-4jp9z"]
Nov 02 16:17:27.646510 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:17:27.646482    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41e2ea12_b95d_47b4_bccd_d1a1fc88bd90.slice/crio-52a7d8e36bf6fd7fb010c40964e99a8579ec37e783f234bddd04a5bf9db3c726 WatchSource:0}: Error finding container 52a7d8e36bf6fd7fb010c40964e99a8579ec37e783f234bddd04a5bf9db3c726: Status 404 returned error can't find the container with id 52a7d8e36bf6fd7fb010c40964e99a8579ec37e783f234bddd04a5bf9db3c726
Nov 02 16:17:27.648486 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.648456    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 16:17:27.756613 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:27.756580    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/must-gather-4jp9z" event={"ID":"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90","Type":"ContainerStarted","Data":"52a7d8e36bf6fd7fb010c40964e99a8579ec37e783f234bddd04a5bf9db3c726"}
Nov 02 16:17:28.765348 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:28.765303    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/must-gather-4jp9z" event={"ID":"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90","Type":"ContainerStarted","Data":"4d2dcd823990c6c7d9d48112e52984f831d5f4c09940d7adb056fe450f4a56ee"}
Nov 02 16:17:28.765348 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:28.765356    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/must-gather-4jp9z" event={"ID":"41e2ea12-b95d-47b4-bccd-d1a1fc88bd90","Type":"ContainerStarted","Data":"23c3413f2a0e6f0f24e6edde29bc841b161b6762b4c350f5ca5f133e904d71b9"}
Nov 02 16:17:28.803459 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:28.803410    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xl725/must-gather-4jp9z" podStartSLOduration=1.04519117 podStartE2EDuration="1.80339558s" totalImagesPullingTime="758.20441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:17:28.800598439 +0000 UTC m=+4901.058667889" watchObservedRunningTime="2026-11-02 16:17:28.80339558 +0000 UTC m=+4901.061465029"
Nov 02 16:17:31.272607 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:31.272579    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gc6sk_453f0247-70c9-4b6c-9272-afe056657e5d/global-pull-secret-syncer/0.log"
Nov 02 16:17:31.819292 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:31.819265    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bt2l4_60554d74-3158-455b-bfef-2d8b4e9e6feb/connection-checker/0.log"
Nov 02 16:17:31.938690 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:31.938662    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dhlcp_bca16489-34d0-4230-b21a-7ea110ff8c7f/connection-checker/0.log"
Nov 02 16:17:32.056733 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:32.056709    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zp66b_f54d3276-9a6f-498e-84ff-a48b1651bc09/connection-checker/0.log"
Nov 02 16:17:32.474250 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:32.474225    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-svzrl_a6432dfb-7757-4a30-a440-8cc647505f76/konnectivity-agent/0.log"
Nov 02 16:17:32.697903 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:32.697861    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-143.ec2.internal_15b2f7b139a98a51bb6e53a9a5605774/haproxy/0.log"
Nov 02 16:17:36.768174 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:17:36.768136    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:17:49.748179 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.748132    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"]
Nov 02 16:17:49.755969 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.755943    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.774237 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.774207    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"]
Nov 02 16:17:49.824102 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.824062    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphwh\" (UniqueName: \"kubernetes.io/projected/2e5fac0a-df99-4059-89bb-0c22da9bac6d-kube-api-access-sphwh\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.824396 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.824358    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-lib-modules\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.824593 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.824573    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-proc\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.824731 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.824717    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-sys\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.824870 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.824856    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-podres\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925579 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925545    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-lib-modules\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925600    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-proc\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925629    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-sys\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925655    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-podres\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925687    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-lib-modules\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925686    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-proc\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.925774 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925736    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-sys\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.926092 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925818    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphwh\" (UniqueName: \"kubernetes.io/projected/2e5fac0a-df99-4059-89bb-0c22da9bac6d-kube-api-access-sphwh\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.926092 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.925739    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5fac0a-df99-4059-89bb-0c22da9bac6d-podres\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:49.947552 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:49.947520    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphwh\" (UniqueName: \"kubernetes.io/projected/2e5fac0a-df99-4059-89bb-0c22da9bac6d-kube-api-access-sphwh\") pod \"perf-node-gather-daemonset-dbdkq\" (UID: \"2e5fac0a-df99-4059-89bb-0c22da9bac6d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:50.070710 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:50.070636    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:50.377849 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:50.377817    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"]
Nov 02 16:17:50.379741 ip-10-0-133-143 kubenswrapper[5041]: W1102 16:17:50.379705    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e5fac0a_df99_4059_89bb_0c22da9bac6d.slice/crio-1efcd7fc11fa487706de745feb33a585373d8044aa70a8792a21544fdf1f8832 WatchSource:0}: Error finding container 1efcd7fc11fa487706de745feb33a585373d8044aa70a8792a21544fdf1f8832: Status 404 returned error can't find the container with id 1efcd7fc11fa487706de745feb33a585373d8044aa70a8792a21544fdf1f8832
Nov 02 16:17:50.949441 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:50.949404    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq" event={"ID":"2e5fac0a-df99-4059-89bb-0c22da9bac6d","Type":"ContainerStarted","Data":"b359146c3091378869f282bff7993634b3b95f8e7cf6d60b062a82f8e32c01f1"}
Nov 02 16:17:50.949850 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:50.949462    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq" event={"ID":"2e5fac0a-df99-4059-89bb-0c22da9bac6d","Type":"ContainerStarted","Data":"1efcd7fc11fa487706de745feb33a585373d8044aa70a8792a21544fdf1f8832"}
Nov 02 16:17:50.949850 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:50.949492    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:17:50.984050 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:50.984001    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq" podStartSLOduration=1.983985035 podStartE2EDuration="1.983985035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:17:50.983788416 +0000 UTC m=+4923.241857866" watchObservedRunningTime="2026-11-02 16:17:50.983985035 +0000 UTC m=+4923.242054501"
Nov 02 16:17:56.964848 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:17:56.964816    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dbdkq"
Nov 02 16:18:06.773412 ip-10-0-133-143 kubenswrapper[5041]: E1102 16:18:06.773345    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rzg46" containerName="extractor"
Nov 02 16:18:16.599901 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:16.599858    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f469495f-npwt9_9ed68aba-0ac0-4d2a-9ff1-4b3302f13abf/kube-auth-proxy/0.log"
Nov 02 16:18:16.921617 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:16.921587    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-jx4lc_67aac315-0378-4703-bd0b-67c163800196/istio-proxy/0.log"
Nov 02 16:18:16.985652 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:16.985611    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k4x2d_23e8c095-a07b-4484-9fdf-0c87e5392cb8/istio-proxy/0.log"
Nov 02 16:18:17.052170 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:17.052138    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-khzpx_97ddb9f7-6726-49f3-9acc-c5602bcf68d1/istio-proxy/0.log"
Nov 02 16:18:17.116267 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:17.116230    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-rdl5t_03da44d6-bcfd-4e9f-88d7-a9b27157d56b/istio-proxy/0.log"
Nov 02 16:18:17.372500 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:17.372411    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-s4wx8_44bb49df-a7f6-426b-812b-df7db3361522/payload-pre-processing/0.log"
Nov 02 16:18:17.504226 ip-10-0-133-143 kubenswrapper[5041]: I1102 16:18:17.504197    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659675459f-cp5t4_e6beba97-264e-4107-bafa-57b5380ad218/router/0.log"